Re: [gentoo-project] Re: [gentoo-dev] [RFC] Splitting developer-oriented and expert user mailing lists

[Gerion Entrup]

Am Sonntag, 3. Dezember 2017, 22:43:19 CET schrieb Michał Górny:
> W dniu nie, 03.12.2017 o godzinie 21∶30 +0100, użytkownik Dirkjan
> Ochtman napisał:
> > On Sun, Dec 3, 2017 at 12:18 AM, Michał Górny  wrote:
> > 
> > > 1. Posting to gentoo-dev@ and gentoo-project@ mailing lists will be
> > > initially restricted to active Gentoo developers.
> > > 
> > > 1a. Subscription (reading) and archives will still be open.
> > > 
> > > 1b. Active Gentoo contributors will be able to obtain posting access
> > > upon being vouched for by an active Gentoo developer.
> > > 
> > 
> > On the face of it, I like this proposal. On the other hand, wouldn't it be
> > better if we just had more active list moderators? That is, moderators who
> > move problematic user's posts to moderated by default, and then withhold
> > the specific posts if necessary?
> 
> I don't think this is really technically feasible. I don't know if mlmmj
> has the specific feature you're asking for, and even if it did,
> moderation with mlmmj is practically impossible to use. Even for low-
> traffic channel like gentoo-dev-announce@ it's not working well.
> 
> > 
> > 2. A new mailing list 'gentoo-expert' will be formed to provide
> > > a discussion medium for expert Gentoo users and developers.
> > > 
> > > 2a. gentoo-expert will have open posting access like gentoo-dev has now.
> > > 
> > 
> > I'm not sure this will be worth it. Who exactly do you think is the
> > audience for this mailing list? What is the goal? How is it different from
> > existing mailing lists?
> 
> The audience is expect users who usually don't need basic support
> but instead want to discuss the development of Gentoo and want to have
> some impact on where it goes.
> 
> The main goal is to be able to restore more developers to gentoo-dev@,
> and be able to focus it on feedback and reviews.
> 
> In other words, the goal is that if the attitude on gentoo-expert
> becomes impossible to bear, the developers can unsubscribe from that
> list without actually losing the ability to give feedback on important
> Gentoo issues.
If core Gentoo developers don't read the expert list, I'm not seeing a high
value in such a list.

I'm a long term Gentoo user, but have read this list a few month only, so
correct me, if I'm wrong. I've seen the main usage of this list in three
aspects:
1. Review and discussion of new (technical) features (eclasses, EAPI, package
manager specs).
2. Information about unmaintained packages.
3. Input and proposals from users.

Splitting the list would reduce the meaning of gentoo-dev to the first point.
The second point has to be handled on the expert list (or both lists), so
proxy maintainers can reply. The third point can only be handled on the expert
list, but core developers have to read it, otherwise the whole point would be
meaningless.

In other projects with similar problems but the technical possibility to 
moderate
some "code of conduct" was adopted, so moderators can ban users on that base
for a fixed amount of time.

Gerion


> 
> > 
> > Cheers,
> > 
> > Dirkjan
> 
> 



signature.asc
Description: This is a digitally signed message part.

[gentoo-dev] [RFC] multiversion ebuilds

[Gerion Entrup]

Hi,

just an idea for now. But what you think about multiversion ebuilds?
Technically this could be realized with the following line in the ebuild itself:
```
VERSIONS=( 3.0.11 3.0.12 3.1 )
```

and the filename without version:
//.ebuild

together with this set of rules:
1. If there is an ebuild with had a version in its name, this ebuild is 
preferred.
 e.g. is a tree consists of "foobar/foobar-1.1.ebuild" and 
"foobar/foobar.ebuild" for version 1.1 the specific ebuild is taken.
2. If the ebuild has the variable VERSIONS specified but also a version in its 
name, the version in its name is taken.
3. There can be only one multiversioned ebuild per package.

Different version keywording can be done as before:
```
if [[ ${PV} == "3.1" ]] ; then
KEYWORDS="~amd64 ~x86"
else
KEYWORDS="amd64 x86"
fi
```

The resolution of versions can be done as before, with the difference that one 
ebuild can represent multiple versions.

The "ebuild" tool needs some adjustments. Maybe it tries to download and build 
all version per default and has an additional flag to specify a single version.

The advantages of this idea I see are:
- Ebuilds are written in a multiversion manner anyway, and then get copied (or 
linked?), so it can be made explicit.
- The diffs between different versions of ebuilds and the commit history are 
way more readable.
- The size of the tree reduces.

Regards,
Gerion



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [RFC] multiversion ebuilds

[Gerion Entrup]

Am Samstag, 12. Mai 2018, 16:21:26 CEST schrieb Ulrich Mueller:
> >>>>> On Sat, 12 May 2018, Gerion Entrup wrote:
> 
> > just an idea for now. But what you think about multiversion ebuilds?
> > Technically this could be realized with the following line in the
> > ebuild itself:
> > ```
> > VERSIONS=( 3.0.11 3.0.12 3.1 )
> > ```
> 
> > [...]
> 
> > The advantages of this idea I see are:
> > - Ebuilds are written in a multiversion manner anyway, and then get
> > copied (or linked?), so it can be made explicit.
> > - The diffs between different versions of ebuilds and the commit
> > history are way more readable.
> 
> That depends on the options you specify for git diff (e.g.,
> --find-renames, --find-copies, or --find-copies-harder).
This does not apply to the online diffs (see e.g. [1]). I assume that most 
users don't have
the git repository checked out.

> 
> > - The size of the tree reduces.
> 
> I very much doubt that (or at least it remains to be proven).
> 
> Currently, when an ebuild is copied for a version bump, it will reuse
> the same blob in the Git repository. With the scheme above, you would
> have to modify the ebuild, which would add a new blob for every
> version bump.
You are right, I've not thought about that. However, this is only true for
the repository not for the rsync copy most(?) users have and not for a checkout.

Gerion

[1] 
https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=d9397ab8d48feb4b1360be614da35fa2faae44d9



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [RFC] multiversion ebuilds

[Gerion Entrup]

Am Samstag, 12. Mai 2018, 15:47:57 CEST schrieb Rich Freeman:
> On Sat, May 12, 2018 at 8:20 AM Gerion Entrup <gerion.ent...@flump.de>
> wrote:
> 
> 
> > Different version keywording can be done as before:
> > ```
> > if [[ ${PV} == "3.1" ]] ; then
> >  KEYWORDS="~amd64 ~x86"
> > else
> >  KEYWORDS="amd64 x86"
> > fi
> > ```
> 
>  From a readability standpoint I could see this getting out of hand if we
> aren't careful. 
I have kind of copied this code from a package, where a  ebuild exists. So 
it is already there.

> Also, we'd want to keep this simple enough that the simple
> act of stabilizing a package doesn't introduce regressions (perhaps even in
> versions that we don't intend to touch).  We'd also need rules about
> conditionals so that the metadata cache works.
Maybe this rule can be added:
4. Stable ebuilds are never multiversioned.

So once an ebuild get stable it is split out of the multiversion ebuild (aka 
copied) and not touched anymore.
Maybe this rule can be smoothed by moving the split to the point where new 
features are introduced in the ebuild:
4. If a multiversion ebuild contains a stable version and is changed 
featurewise (e.g. a new useflag), then the stable version is split out of the 
multiversion ebuild.

I don't know the internals of the metadata cache, so I can't say something to 
that.


> You also would need to be careful about ebuild revisions in general, since
> you're potentially affecting multiple versions at the same time.
You are right. But maybe it's the same fix, that is applied to all versions at 
once.

> How would revbumps fit into this?  All kinds of bad things can happen by
> editing ebuilds in place, and this would compound the issue.
Maybe it is possible to solve with the following method:
1. If the fix affects all ebuild versions, revbump all versions in the ebuild.
2. If the fix affects only one version, split out this version of the ebuild 
and make it an explicit one (together with the fix).

The thing is that it is an additional concept. It does not replace the current 
mechanisms and at best reduces overhead for ebuilds that are identical.

Gerion


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [RFC] multiversion ebuilds

[Gerion Entrup]

Am Samstag, 12. Mai 2018, 16:24:00 CEST schrieb R0b0t1:
> On Sat, May 12, 2018 at 7:20 AM, Gerion Entrup <gerion.ent...@flump.de> wrote:
> > - The size of the tree reduces.
> >
> 
> If this is a big concern you may be able to mount the portage tree
> under a compressed loopback filesystem. It may even be worth
> considering that as a recommended-by-handbook default as the benefit
> for compressing the ebuilds is likely huge anyway.
This seems reasonable. However, first duplicating code and then deduplicate it 
at file system level seems weird.

Gerion


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [RFC] multiversion ebuilds

[Gerion Entrup]

Am Mittwoch, 16. Mai 2018, 09:38:30 CEST schrieb Michał Górny:
> W dniu sob, 12.05.2018 o godzinie 14∶20 +0200, użytkownik Gerion Entrup
> napisał:
> > Hi,
> > 
> > just an idea for now. But what you think about multiversion ebuilds?
> > Technically this could be realized with the following line in the ebuild 
> > itself:
> > ```
> > VERSIONS=( 3.0.11 3.0.12 3.1 )
> > ```
> > 
> > and the filename without version:
> > //.ebuild
> > 
> > together with this set of rules:
> > 1. If there is an ebuild with had a version in its name, this ebuild is 
> > preferred.
> >  e.g. is a tree consists of "foobar/foobar-1.1.ebuild" and 
> > "foobar/foobar.ebuild" for version 1.1 the specific ebuild is taken.
> > 2. If the ebuild has the variable VERSIONS specified but also a version in 
> > its name, the version in its name is taken.
> > 3. There can be only one multiversioned ebuild per package.
> > 
> > Different version keywording can be done as before:
> > ```
> > if [[ ${PV} == "3.1" ]] ; then
> > KEYWORDS="~amd64 ~x86"
> > else
> > KEYWORDS="amd64 x86"
> > fi
> > ```
> > 
> > The resolution of versions can be done as before, with the difference that 
> > one ebuild can represent multiple versions.
> > 
> > The "ebuild" tool needs some adjustments. Maybe it tries to download and 
> > build all version per default and has an additional flag to specify a 
> > single version.
> > 
> > The advantages of this idea I see are:
> > - Ebuilds are written in a multiversion manner anyway, and then get copied 
> > (or linked?), so it can be made explicit.
> > - The diffs between different versions of ebuilds and the commit history 
> > are way more readable.
> > - The size of the tree reduces.
> > 
> 
> In my opinion, this puts more effort into inventing a problem than
> solving one.  In order to consider a particular idea thouroughly, you
> should also consider potential disadvantages, rather than focusing
> purely on advantages as you see them.
> 
> While one might think that this will help developers, it is going to be
> a maintenance nightmare.  Just compare the workflow.
> 
> Currently, adding a patch affecting runtime involves copying the ebuild,
> and applying the patch.  Later on, the old revision is just removed. 
> With your solution, you need to adjust VERSIONS, add conditional; later
> on, you need to adjust VERSIONS, find all conditionals, remove them. 
> Not only it involves more work but also increases the risk of accidental
> breakage.
I'm not sure, if I understand you correctly, so I try to express myself:
If an ebuild (say foobar-1.0.ebuild) needs a patch affecting runtime, the
current workflow is to copy the ebuild, add the code that applies the patch
and do a revbump (to foobar-1.0-r1.ebuild). The old ebuild remains
untouched.
With the VERSIONS approach, the VERSIONS need to be adjusted to contain
the revbump version as well, and add a conditional that applies only in the
revbump version the patch and otherwise not.

The VERSIONS approach does not break the old mechanism. So if a patch
needs to be applied, the multiversion ebuild (that contains other versions
as well, say foobar.ebuild with VERSIONS="1.0 1.1 1.2") can just be copied,
renamed with the revbump number (to foobar-1.0-r1.ebuild) and completed
with the apply line. Once the old versions should be deleted the VERSIONS
variable can be adjusted (to VERSIONS="1.1 1.2").


> The arch team work is going to become a nightmare.  Currently, they just
> use 'ekeyword' tool to mass-edit ebuilds.  With your solution, they're
> now have to find the correct conditional (or add one), and update
> keywords there.  I can already imagine monsters like:
> 
>   if pv1; then
> KEYWORDS="~amd64"
>   elif pv2; then
> KEYWORDS="amd64 ~arm64 x86"
>   elif pv3; then
> KEYWORDS="~amd64 ~arm64 ~x86"
>   elif pv4; then
> KEYWORDS="amd64 ~arm64 ~x86"
>   fi
> 
> Basically, any action requiring >1 arch team would involve creating
> a new version at least temporarily.  And I seriously doubt arch teams
> would really want to spend time collapsing those afterwards.
You are right, I have not thought about that. Maybe the approach of R0b0t1
would work.


> The algorithm you presented might look nice at first.  But remember that
> the developers will now have to do the same thing -- i.e. discover which
> ebuild should they look at.  You're moving for a clear 1 ebuild : 1 file
> mapping to N ebuilds : 1 file.

Gerion

Re: [gentoo-dev] Re: rfc: why are we still distributing the portage tree via rsync?

[Gerion Entrup]

Am Donnerstag, 5. Juli 2018, 14:03:36 CEST schrieb Martin Vaeth:
> Matt Turner  wrote:
> > The ebuild tree is 600MB with rsync and cannot fit on the partition
> > with git.
> >
> > I'd be happy to switch if the space requirements were similar.
> 
> If one git repacks every few syncs one needs currently about 800 MB.
> 
> With additionally squashfs (zstd) (+ overlayfs) the full
> archive size is currently <600 MB.
> 
> In both cases, the temporary disk space is slightly more, of course.
> For a 1GB reserved partition I'd use the partition for the temporary
> mounting and store the archive somewhere else, but I think chances are
> good that you also come through with only a git repack after
> every sync. A difficulty might be the very first git sync.

Would it possible to take the bare repo (< 600 MB) and only mount the latest 
checkout (with fuse eg)?

Gerion

Re: [gentoo-dev] Packages up for grabs: net-misc/sobby, net-libs/obby, net-libs/libinfinity, net-libs/net6

[Gerion Entrup]

Am Dienstag, 27. November 2018, 13:47:25 CET schrieb Ulrich Mueller:
> > On Mon, 26 Nov 2018, Tiziano Müller wrote:
> 
> > the following packages are up for grabs since I have no use for them
> > anymore:
> 
> > net-misc/sobby
> > net-libs/obby
> > net-libs/libinfinity
> > net-libs/net6
> 
> > They are unfortunately not up-to-date, and 3 minor bugs are open for
> > net-libs/libinfinity:
> 
> > https://bugs.gentoo.org/527158
> > https://bugs.gentoo.org/537488
> > https://bugs.gentoo.org/670610
> 
> > Upstream seems to be still around for the editor itself (gobby, not in
> > the tree anymore).
> 
> > If nobody picks them up, they are likely to get tree-cleaned since all
> > except one are libraries nobody seems to be using.
> 
> Actually, sobby is the server counterpart for app-emacs/rudel. So unless
> anyone else is interested, the Emacs team will take these three:
> 
>net-misc/sobby
>net-libs/obby
>net-libs/net6
> 
> Ulrich

It seems that sobby was deprecated by the Gobby team (also last commit
from 2012). Libinifity also provides an obby server (infinoted). Does
app-emacs/rudel run with infinoted as well?

Gerion



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [RFC] New QA policy: Packages must not disable installing manpages via USE flags

[Gerion Entrup]

Am Dienstag, 23. Juli 2019, 04:00:07 CEST schrieb Kent Fredric:
> On Mon, 22 Jul 2019 21:08:51 -0400
> Aaron Bauman  wrote:
> 
> > 1. I want some documentation
> > 2. It doesn't ship from upstream (without crazy extra deps)
> > 3. Gentoo guy hooked me up and packaged it pre-built with it
> > 4. Thanks!
> 
> The proposal as-stated is:
> 
> 1. Documentation requires even 1 additional dep
> 2. Thou may not use a USE flag for this
> 3. Thus, if you want to elide the dependency from *any* merge graph,
>you must elide it from *all* merge graphs.
> 4. Thus, you must locally perform some non-standard hackery that will
>be different for every package to produce these, work out where to put
>it which is also not standardised, and also prohibit the user from
>being able to update these themselves via a revision bump, _AND_ you
>will need to put in place non-standard mechanisms to ensure it gets
>updated when you update the package, in order for the documentation
>not to diverge from the sources.
What about a compromise?:
Deliver a (prebuild) manpage as package maintainer by default, but keep
a use flag "man-build" (or whatever) that builds the man page for everyone
(also the maintainer herself)  with use of the crazy extra deps. So a user can
do (incomplete) version bumps and gets a manpage and the maintainer
gets the prebuild manpage in a defined way.


> 
> There's a lot of "U, thats bad" in point 4.
> 
> Hence, counter-proposals are trying to look at a way to achieve points
> 2 & 3 in your list, without resorting to barbaric torture and inherent
> fragility.
> 
> We understand the /achieve, but the mechanism proposed doesn't suit, as
> stated.

Regards,
Gerion 



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Last rites: app-text/pdfshuffler

[Gerion Entrup]

Am Sonntag, 1. Dezember 2019, 21:09:50 CET schrieb Michał Górny:
> # Michał Górny  (2019-12-01)
> # Unmaintained.  Last release in 2012.  Buggy ebuild.
> # Removal in 30 days.  Bug #658302.
> app-text/pdfshuffler

This is also Python 2 only and GTK 2 (with very experimental GTK 3 support).
However, there seems to be a modern (active) fork with Python 3 and
GTK 3:
https://github.com/jeromerobert/pdfarranger

Gerion

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Needs ideas: Upcoming circular dependency: expat <> CMake

[Gerion Entrup]

Am Donnerstag, 19. Dezember 2019, 19:43:37 CET schrieb Sebastian Pipping:
> On 19.12.19 18:37, Michał Górny wrote:
> > We have a better alternative that lets us limit the impact on the users.
> > Why not use it?
> 
> Which one?  The CMake bootstrap copy?  The adding to stage3 one?

Is it possible to show a specific error message when running in a
circular dependency?

When I get it right, the problem occurs only one time when solved
with a bundled use flag. As soon as expat and/or cmake are installed,
follow up versions can be merge without problems.

If an error message can be shown, maybe this is enough as a hint:
"expat and cmake have circular dependencies. Emerge it the first time with:
USE=bundled-cmake emerge -1 cmake expat
and then just don't care anymore about this use flag."

Of course, the same applies for other known circular dependencies, too.
At least for me as a user, this would be enough.

Best,
Gerion


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Last rites: dev-python/* leaf packages

[Gerion Entrup]

Am Donnerstag, 5. Dezember 2019, 01:15:48 CET schrieb Aaron Bauman:
> dev-python/pycdio

Has Python 3 support since 2.1 (released in August this year). Developed by 
libcdio itself.

> app-text/pdfshuffler

Was last rited a few day ago. As I said, pdfarranger 
(https://github.com/jeromerobert/pdfarranger) seems to be the modern follow up 
project.

Gerion




signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Ebuild Generators

[Gerion Entrup]

Am Dienstag, 11. Februar 2020, 16:18:44 CET schrieb Tom Gillespie:
> For historical curiosity there was also
> https://github.com/domenkozar/g-pypi at one point (similar to
> https://github.com/rafaelmartins/g-octave). Having used g-octave, the
> primary issue is as Michał says, there are a lot of corner cases that
> the generation doesn't handle correctly which lead to broken ebuilds
> and maintenance headaches. Speaking for python, catching and
> correcting the use of setup_requires and other insanity visited upon
> us by the wonders of setuptools makes this a non-starter
That's essentially my whole point. Generators are _not_ able to catch all
corner cases but should be able to lower the initial ebuild creation
cost. That could be such things like figure out the license, try to
fill the source url, 
In principal: Try to automate what is possible and leave the rest to
the maintainer.

But in that state they are a tool for the Gentoo developer not for the
user.

Gerion



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] GSoC 2020: Call for mentors and project ideas

[Gerion Entrup]

Am Sonntag, 2. Februar 2020, 12:47:55 CET schrieb Benda Xu:
> Dear Fellows,
> 
> alicef  writes:
> 
> > As always, Gentoo plans to participate in the Google Summer of Code
> > 2020. We are looking for new project ideas and are always open for
> > new mentors.
> > Google Summer of Code is a big opportunity for making Gentoo project more
> > visible and get more people interested to join Gentoo and helping out.
> >
> > [...]
> 
> This year's GSoC organization application deadline is on Feb 5.  The
> more project ideas, the better Gentoo will show itself to be prepared.
> If you have been thinking of adding projects to our GSoC 2020 list, this
> is a good chance to do so.
> 
> Cheers,
> Benda
> 

Hi,

I saw the idea „Big Data Infrastructure by Gentoo“ and found it kind of
interesting. However, I have a little bit the fear that a full automation
won't be possible and the whole project becomes a little bit like g-sorcery
(gs-pypi, gs-elpa) or g-octave: a really cool project but not used at a
large scale.

What do you think of the idea to not do this fully automated but supervised
by a maintainer? With that I mean an ebuild generator that generates only
the parts of the ebuild that it can easily parse and then present the ebuild
draft to a maintainer who completes it to an full ebuild. As far a I know no
tool like this exists. I think the focus shift helps a lot:
Developing a tool for the Gentoo maintainer not the Gentoo user.

I'm only "maintaining" an overlay so maybe I'm  missing experience
but I often have wished a tool that automatically parses the language specific
packaging files and is able to generate a primitive ebuild out of that.
Maybe it even can do this in an interactive way:
"Hey, upstream needs the dependency 'foo'. In the Gentoo packages I have found
'dev-bar/foo' and 'dev-util/foo'. What is the correct one?"

With a not fully automatic tool also packages can be parsed that are not
in a complete closed ecosystem, like a 'meson.build' file or cmake files for
C++/C programs. But of course package databases like Maven/Cargo/Pypi are
also candidates.

Unfortunately, I have no time currently to participate in the GSOC. I just
want to mention this here as an idea. Please comment or correct me, if
such a tool already exists.

Best,
Gerion



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Ebuild Generators (Was: GSoC 2020: Call for mentors and project ideas)

[Gerion Entrup]

Am Montag, 3. Februar 2020, 05:20:42 CET schrieb Benda Xu:
> Gerion Entrup  writes:
> 
> > I saw the idea „Big Data Infrastructure by Gentoo“ and found it kind of
> > interesting. However, I have a little bit the fear that a full automation
> > won't be possible and the whole project becomes a little bit like g-sorcery
> > (gs-pypi, gs-elpa) or g-octave: a really cool project but not used at a
> > large scale.
> 
> Yes, that's true.  I share the same observation and concern with you.
> 
> This is one exception: the CRAN ebuild generator powered R overlay has
> been running well for 8 years.
> 
>   https://wiki.gentoo.org/wiki/Project:Science/Overlay/R
Hmm, interesting, thank you.


> > What do you think of the idea to not do this fully automated but supervised
> > by a maintainer? With that I mean an ebuild generator that generates only
> > the parts of the ebuild that it can easily parse and then present the ebuild
> > draft to a maintainer who completes it to an full ebuild. As far a I know no
> > tool like this exists. I think the focus shift helps a lot:
> > Developing a tool for the Gentoo maintainer not the Gentoo user.
> 
> Yes, that makes a lot of sense.  The R overlay follows this model.  Most
> of the ebuilds are automated.  When an ebuild generation fails, we add
> the ebuild manually, understand it and then update the generator to
> cover it in the future.
Is this possible in all cases? I think of adding custom patches,
appropriate mapping of dependencies, check for things like desktop icon
cache...


> > I'm only "maintaining" an overlay so maybe I'm  missing experience
> > but I often have wished a tool that automatically parses the language 
> > specific
> > packaging files and is able to generate a primitive ebuild out of that.
> > Maybe it even can do this in an interactive way:
> > "Hey, upstream needs the dependency 'foo'. In the Gentoo packages I have 
> > found
> > 'dev-bar/foo' and 'dev-util/foo'. What is the correct one?"
> 
> Yes, that's the way R overlay is working.  And I have a similar plan and
> proof-of-concept solution for the Java Maven overlay.
Nice to hear. I think, it is meaningful to solve all generation with one
tool. Maybe it can even "recognize" the used build system and package
database. Is this your plan, too?


Best,
Gerion


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Ebuild Generators

[Gerion Entrup]

Am Montag, 3. Februar 2020, 13:19:52 CET schrieb Benda Xu:
> Hi Gerion,
> 
> Gerion Entrup  writes:
> 
> >> Yes, that makes a lot of sense.  The R overlay follows this model.  Most
> >> of the ebuilds are automated.  When an ebuild generation fails, we add
> >> the ebuild manually, understand it and then update the generator to
> >> cover it in the future.
> >
> > Is this possible in all cases? I think of adding custom patches,
> > appropriate mapping of dependencies, check for things like desktop
> > icon cache...
> 
> That's too complex to handle automatically.  Luckily, in R overlay, such
> packages are less than 5%.  An ebuild generator is based on the
> observation that many language-specific packages are trivial to fetch,
> compile and install.
> 
> >> > I'm only "maintaining" an overlay so maybe I'm  missing experience
> >> > but I often have wished a tool that automatically parses the language 
> >> > specific
> >> > packaging files and is able to generate a primitive ebuild out of that.
> >> > Maybe it even can do this in an interactive way:
> >> > "Hey, upstream needs the dependency 'foo'. In the Gentoo packages I have 
> >> > found
> >> > 'dev-bar/foo' and 'dev-util/foo'. What is the correct one?"
> >> 
> >> Yes, that's the way R overlay is working.  And I have a similar plan and
> >> proof-of-concept solution for the Java Maven overlay.
> >
> > Nice to hear. I think, it is meaningful to solve all generation with one
> > tool. Maybe it can even "recognize" the used build system and package
> > database. Is this your plan, too?
> 
> No, I don't think it possible as far as I can see...  That would be a
> strong AI.
I mean only on a primitive base:
```
if link contains "pypi":
# it's a Python package from pypi
handle_pypi()
elif work_tree contains "setup.py":
# it's a Python package
handle_generic_python()
elif work_tree contains "meson.build":
handle_meson_package()
...
```

Best,
Gerion


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Packaging changes in LLVM 10

[Gerion Entrup]

Am Montag, 16. März 2020, 09:23:49 CET schrieb Michał Górny:
> Hi,
> 
> Since 10.0.0 final is around the corner, I'd like to take a minute to
> inform developers of packaging changes in Gentoo that affect its
> revdeps.  Following frequently repeated requests from Gentoo developers
> and users, LLVM 10 is stopping to use BUILD_SHARED_LIBS=ON, i.e. split
> shared libraries.  Instead, we'll be using the 'dylib' model recommended
> upstream.
> 
> This means that sys-devel/{llvm,clang} no longer install those small
> libLLVM*.so and libclang*.so files.  Instead, one big libLLVM-${ver}.so
> and libclang-cpp.so.${sover} are installed (yes, I know this historical
> inconsistency sucks).  Also note that Clang continues installing
> libclang.so which is a *C* (not C++) library.
> 
> Most of LLVM revdeps should not have problem with LLVM dylib, as it was
> the de-facto standard on other distros.  llvm-config handles it
> transparently.  If you're dealing with a custom build system that
> doesn't handle it, it's as simple as trying '-lLLVM' first (note that
> you will '-L' due to slotting).
> 
> Clang revdeps may have trouble with clang-cpp library since it's
> a fairly recent addition.  Again, the solution is simple: '-lclang-cpp'
> (repeating: do not confuse it with -lclang which is a different library
> for historical reasons).
> 
> sys-devel/lld does not install any libraries anymore.  If you ever need
> them, please ping me and we'll try to come up with something upstream.
> 
> dev-util/lldb installs liblldb.so.  However, I'm not familiar with that
> library and I don't know if it has any consumers.
> 
> If you need any help, please don't hesitate to ping me.

Hi,

when I compile LLVM for myself, I also always choose SHARED_LIBS simply
because of RAM usage when linking the libraries. In the past, I was not
able to link the single library on my 16 GB machine (I have not tested
it now and this could be dependent on other circumstances). What is the
current expected RAM usage for LLVM 10?

Best,
Gerion


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [gentoostats continued] Collected data and justification for it

[Gerion Entrup]

Am Donnerstag, 7. Mai 2020, 09:29:36 CEST schrieb Michał Górny:
> I'm going to start with the data and uses I can think of.  Please reply
> with other things you can think of.
> 
> 
> 1) list of selected packages (@world)
> 
> We would use this to determine the popularity of individual packages,
> plus by scanning their dependencies we would be able to make combined
> statistics for direct usage + dependencies of other selected packages. 
> This would allow us to judge which packages need more of our attention.
> 
> For example, as we port Python packages to Python 3.8 the packages with
> more declared users would be ported first.

You may want to collect packages installed per sets, too.
I mainly do what Hans mentioned in his mail to this thread but with sets.
For example I have a KDE-PIM set that installs only my needed subset of
the KDE PIM suite. (I also use this as common workaround for yet missing
runtime dependencies / suggestions made by pkg_postinst.)

Retrieval of this packages would be straight forward: Look at world_sets
and collect all packages that are installed by the set.

 
> 2) USE flags on installed packages (disabled/default/enabled)
> 
> This would allow us to determine which flags users are most likely to
> actually rely on.  This could determine tested flag combinations,
> defaults, and required level of support for individual flags.
> 
> For example, if OCaml bindings on some package are broken and require
> a lot of work, I would find useful to know how likely it is that anyone
> is using it.  Or if a lot of people are enabling 'frobnicate' flag,
> I could consider employing USE defaults.

I'm not sure, if Portage is capable of this, but a distinction in USE
flags needed to fulfil some dependency of another package and USE flags
actively activated by the user could be useful.

Dependency use flags should be treated with a higher priority in my
opinion, since they enable the installation of another package (tree),
while use flags that enable a certain feature that is not used elsewhere
are more "nice to have".


Best
Gerion


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] rfc: noarch keyword

[Gerion Entrup]

Am Donnerstag, 19. März 2020, 02:59:56 CET schrieb Kent Fredric:
> On Wed, 18 Mar 2020 17:52:25 +
> James Le Cuirot  wrote:
> 
> > Not quite. Tools like repoman will need to be updated to understand
> > that an ebuild with KEYWORDS="amd64" can depend on another ebuild with
> > only KEYWORDS="noarch". I do think the idea has merit though.
> 
> But the inverse is _not_ true, an ebuild with KEYWORDS="noarch"
> *cannot* depend on another ebuild with only KEYWORDS="amd64".
Maybe I misunderstand something but shouldn't that be the normal case?
Every single Python package (candidates for noarch) for example depends
on the Python interpreter, which must have non noarch keywords.


> Otherwise it breaks the entire stabilization graph.
The condition would be: If the interpreter is stable for an arch, all
it's interpreted code is also stable for this arch.


Best,
Gerion



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] [kde overlay] Up for grabs: dev-util/arcanist

[Gerion Entrup]

Am Dienstag, 19. Mai 2020, 15:56:53 CEST schrieb Andreas Sturmlechner:
> KDE project has moved away from the unloved Phabricator instance to GitLab 
> (invent.kde.org), so there is no reason for us to keep this package in kde 
> overlay for much longer.
> 
> If for some reason you still rely on kde overlay providing this package, 
> consider picking it up now and maintain it in gentoo.git or your own place.
> 
> Regards,
> Andreas

I use this tool for my work. Is it in a fashion to push it to gentoo.git?
If yes, I can proxy maintain it.

If I get it right, there is not real installation, but only "copy all in opt 
and symlink the binary".
Is is meaningful to make a fake version like arcanist-2020.05.19?

Gerion


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] New eclass: openvdb.eclass

[Gerion Entrup]

Friendly ping (or have I missed any discussion channel?).
As a user, I'm really looking forward to see blender >=2.80 in the tree.
According to https://bugs.gentoo.org/689740 this eclass is the main blocker.

Gerion

Am Mittwoch, 27. Mai 2020, 03:40:04 CEST schrieb Adrian Grigo:
> Hello, I am a proxied maintainer of blender and openvdb and wish to propose a 
> new eclass and USE_EXPAND variable to ensure the same version of the openvdb 
> abi is used when building the blender, openvdb and openimageio packages.
> 
> The tree currently contains openvdb 4 and 5, and I plan to add 6 and 7 soon 
> to support my new blender PR. Each major release of openvdb has a new ABI, 
> and can also support legacy versions. However only one version can be built 
> at a time, which is passed to cmake through OPENVDB_ABI_VERSION_NUMBER, and 
> produces a library with functions named xxx_7abi5 when building openvdb 7 
> with support for abi 5. Client packages like blender and openimageio which 
> depend on openvdb and also need to be compiled with the same version number 
> in order to link correctly.
> 
> To enforce this I propose to add a new USE_EXPAND variable called 
> OPENVDB_ABI, which is set to the version number the user wants to use in the 
> system eg 5, which expands to openvdb_abi_5. The packages can then append 
> -DOPENVDB_ABI_VERSION_NUMBER=${OPENVDB_ABI} to ensure they pass the same 
> value to cmake.
> 
> To reduce the boilerplate code required to ensure that only one supported 
> value is set in OPENVDB_ABI and used by all packages, so I propose to add an 
> eclass to maintain this in one place. Similar to PYTHON_COMPAT, each ebuilds 
> set OPENVDB_COMPAT to specify which legacy ABI they can support. The 
> openvdb.eclass uses it to set global variables OPENVDB_REQUIRED_USE and 
> OPENVDB_SINGLE_USEDEP and verify in pkg_prepare that only one of the 
> openvdb_abi_X flags is set.
> 
> Each ebuild sets REQUIRED_USE="${OPENVDB_REQUIRED_USE}" which evaluates to ^^ 
> ( openvdb_abi_3 openvdb_abi_4 ... ) to ensure only one abi is enabled from 
> those it supports.
> They set RDEPEND="openvdb? ( media-gfx/openvdb[${OPENVDB_SINGLE_USEDEP}] )" 
> which evaluates to openvdb_abi_3(-)?,openvdb_abi_4(-)?,... to allow portage 
> to enforce the same abi version is built between all packages.
> 
> With a new release of openvdb, I will only need to add it to the 
> openvdb_abi.desc file, to an eclass internal list _OPENVDB_ALL_ABI, and to 
> OPENVDB_COMPAT for each ebuild which supports the new version.
> 
> Please review my proposed eclass below. I have tested it works when 
> OPENVDB_ABI is missing, set to invalid values, or set to values that cannot 
> satisfy all package requirements simultaneously, and that the system 
> recompiles all packages when the ABI is changed. It also works when set to 
> appropriate values.
> 
> I considered other alternatives during development. It is possible to 
> manually add the expanded REQUIRED_USE and RDEPEND strings to each package, 
> but with each new version of openvdb and multiple client packages to maintain 
> this becomes messy and error prone. The user also needs to set the same 
> openvdb_abi_X flag for each package. While currently there are only three 
> packages which need to be synchronised, any other package which depends on 
> openvdb in future will benefit from this infrastructure.
> 
> Also without the USE_EXPAND variable the ebuilds do not have access to the 
> version number to pass to the build system, and need to generate it by 
> testing each USE flag. So I think this is the cleanest solution.
> 
> I also looked into slotting openvdb as then packages could specify the 
> slot/subslot required eg openvdb:7/5 but don't think this is possible. This 
> would be a major undertaking as it has a static as well as dynamic core 
> library, another library for python, and the cmake modules that find openvdb 
> use the output of a binary at /usr/bin/vdb_print to determine which version 
> to link against. It would also not centralise the boilerplate code.
> 
> For further information and my discussion during development see 
> https://github.com/redchillipadi/ebuild-overlay/issues/4
> 
> Please let me know if there is a better solution or improvements are required.
> 
> Kind Regards,
> Adrian Grigo
> 
> 
> # Copyright 1999-2020 Gentoo Authors
> # Distributed under the terms of the GNU General Public License v2
> 
> # @ECLASS: openvdb.eclass
> # @MAINTAINER:
> # Adrian Grigo 
> # @ AUTHOR:
> # Author: Adrian Grigo 
> # Based on work of: Michał Górny  and Krzysztof Pawlik 
> 
> # @SUPPORTED_EAPIS: 5 6 7
> # @BLURB: An eclass for OpenVDB to control which ABI version is compiled
> # @DESCRIPTION:
> # The OpenVDB package is a library for sorting and manipulating sparse
> # data structures with dynamic topology as required for use in volume
> # rendering for computer graphics.
> #
> # Each major version of OpenVDB provides an updated ABI, as well as
> # the ability to compile 

Re: [gentoo-dev] Declare the type of source

[Gerion Entrup]

Am Montag, 28. Juni 2021, 16:13:41 CEST schrieb Rich Freeman:
> On Mon, Jun 28, 2021 at 9:46 AM Michael Orlitzky  wrote:
> >
> > On Mon, 2021-06-28 at 15:00 +0200, Agostino Sarubbo wrote:
> > >
> > > Instead, imagine that each ebuild declares a variable called SOURCETYPE ( 
> > > or
> > > similar, or in metadata.xml if you prefer ) and with a tool like 
> > > equery/eix we
> > > are able to get the list of all packages that compiles C code.
> > >
> >
> > I think all you are really asking for is that we stop omitting a random
> > subset of @system from *DEPEND.
> >
> > This is long overdue, for many reasons, but in particular it would
> > force us to declare a dependency on a C compiler if one is needed and
> > allow you to re-test only those packages that use a C compiler.
> 
> ++ - this would also support parallel building of @system.
> 
> Obviously we'll still need a core set of packages needed for
> bootstrapping/etc, but there is no reason @system couldn't just be
> another virtual.
> 
> You could also have convenience virtuals for things like the C
> toolchain and so on.  This will both support alternate implementations
> and avoid having to have laundry lists of deps in every ebuild.
> 
> A simple way to transition would be to create a system virtual and add
> it to all ebuilds, but ask that this be removed in future updates in
> favor of more specific dependencies.  Over time then the tree would
> move to specified true deps.  Catalyst could still use a virtual as a
> target for bootstrapping stages.
> 
> Another tool that would be useful is what some other distros do - use
> mount namespaces/etc to allow build systems to only see parts of the
> filesystem (down to the file level) that are specified in
> dependencies.  This would basically eliminate unspecified or automagic
> dependencies, since anything not specified basically doesn't exist at
> build time.  If you didn't want to use mount namespaces then our
> sandbox already allows limiting read access to only specified files -
> we just configure it to allow read-only to everything for every
> package.
> 
> 

Hello,

I was already writing an answer, which describes basically the same idea,
when Rich's mail arrived. I want to post my mail anyway. Maybe it provides
some additional information:

Wouldn't the right place be in BDEPEND, maybe hidden by some eclass magic?

Some time ago, I have looked into Nix. They try to get reliable input by path
manipulation and therefore can depend on the compiler (with a specific
version). If I get there build system right, it only builds a software if
all dependencies are installed beforehand in a specific input specific
folder.

I'm questioning myself, if this is also possible in a Gentoo compatible
way with Linux namespaces?:
1. Create a new namespace for / (consisting of no files).
2. Bindmount/Link every file of each dependency into it at the exact same place.
3. Link some socket(?) to communicate with the outer portage.
4. Trigger the build process.

I imagine something like the the TemporaryFileSystem feature of Systemd
together with BindPaths [1]. This uses Linux namespaces internally, too.

In a pseudo service file syntax:
```
[Service]
ExecStart=ebuild mytool-1.0.2 compile
TemporaryFileSystem=:ro
BindPaths=$(equery files $(equery depgraph =mytool-1.0.2))
```

This should only build, if _all_ build dependencies are present
(including every compiler and base system tool). Of course, it needs a
bigger rework of the portage build process.

Gerion

[1] 
https://www.freedesktop.org/software/systemd/man/systemd.exec.html#BindPaths=


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Idea: User centric kernel configuration

[Gerion Entrup]

Am Donnerstag, 11. März 2021, 17:41:45 CET schrieb Peter Stuge:
> Hi,
> 
> Gerion Entrup wrote:
> > the Linux kernel has _a lot of_ configuration options, way too many to 
> > configure them by hand.
> 
> I actually disagree strongly with that; I think it's important to
> actively decide what kernels include, and I routinely do, but of
> course not everyone will. I've made sure to include a kernel build
> when teaching systems administration courses and would again.
> 
> As the kernel becomes more complex the threshold for the first
> configuration also rises, but it's still completely possible to learn
> what you need in order to successfully configure your own kernel.
> I guess it's on the order of a weekend project given some basic
> understanding of computer architecture and programming.

I think, we mean two different things here. I for myself also configured
my own kernel(s) for years. I also actively teach students to do so.
However, I have never looked into all 18000 configuration options. I
don't understand them all (by far).

Actually, most of the times, I do a `make localmodconfig`, click through
various subsystems and activate what I think that it is useful or sounds
nice. Then, to update a kernel, I use `make oldconfig` and answer the
questions as good as I can (taking the default otherwise).

If my kernel is not capable to do something (for example run docker
containers), I take a look in the Gentoo Wiki and "copy" the options
into my own config.

But for me, this is not an informed decision. Of course, I learn
something in this process, maybe also more than a precompiled kernel
user, but I have by far not created a minimal config or even begin
to understand all subsystems and different configuration options.

I'm sure that this is similar to most Gentoo users.

BTW, here is a project to create a minimal config (kind of "ideal"
config) based on a "golden run" [1]. However, I have not compared such
a config with my own homegrown config.


> > This requires a mapping between user oriented "features" and the kernel
> > internal configuration options.
> 
> So the challenge here is that the kernel is disjoint from user space,
> and while the kernel API remains stable over time consumer requirements
> such as "docker" or "cryptsetup" will mean different things for
> different versions of particular user space software.
> 
> 
> > Do you think that it is useful and feasible to combine these two mechanisms?
> 
> AFAIK there's no generic method for formal kernel requirements in user
> space packages and there's also no sanctioned method for quering
> kernel capabilities. The only thing available is /proc/config if that
> was enabled in the kernel build, and there are of course reasons to
> leave it out, and it only applies to the particular running kernel,
> e.g. useless for cross-compilation. There, it would be possible to
> read the kernel configuration file if the kernel source code is
> available when the userspace package is being built, but that's not
> guaranteed.
> 
> In Gentoo, linux-info.eclass provides linux_config_exists() to do all
> of this, but order to become a widespread success there would have to
> be one method for upstreams to maintain these requirements as part of
> their packages, rather than forcing the burden on package maintainers
> to repeat the same detective task in every single distribution.
> 
> I think it would be very useful to create something generic for that,
> but that's certainly no small task.
> 
> And realistically I only see it succeeding if Linux Foundation decides
> to push it onto the world.

Sorry, I may have expressed this not clearly. I guess, we actually have
the same opinion here. I don't want to do this mapping automatically. I
don't want a "fully automatic" config or touch any (user space) ebuilds.

My idea is to patch Kconfig (as part of gentoo-sources) to provide the
same "features" (i.e. mappings) that are already present in the Gentoo
wiki.

For example this could result in this installation description for Docker
(within the Wiki):
"To enable kernel support for Docker, enable
'Gentoo Linux -> Support for user space programs -> Docker'"

This Kconfig flag than depends on all options that are needed for proper
Docker support (as already described in the Wiki) like cgroups etc..

This would allow users to configure there kernels the same way as before
but with some additional convenience shortcuts.

Of course, this is a high maintenance burden for the kernel package
maintainers. Therefore, this mechanism maybe can be automated:
Grep the wiki pages for kernel config snippets and automatically
condense that in a Kconfig readable configuration option.


> > A possible way could be to au

[gentoo-dev] Idea: User centric kernel configuration

[Gerion Entrup]

Hi,

the Linux kernel has _a lot of_ configuration options, way too many to 
configure them by hand.
Many of these configuration options seem to be kernel centric (they deactivate 
a specific compilation unit, they deactivate specific source code parts).
However, my main configuration requirements as a user are mostly user space 
program or feature centric. I want to activate kernel support for a specific 
feature, e.g. docker support or cryptsetup support.

This requires a mapping between user oriented "features" and the kernel 
internal configuration options. This mapping exists already at the Gentoo wiki 
[e.g. 1, 2, 3].
Gentoo also provides (via gentoo-sources) a set of patches that adds some meta 
configuration options to enable some of exactly that user specific features 
(Systemd, OpenRC and Portage support).

Do you think that it is useful and feasible to combine these two mechanisms?
A possible way could be to automatically extract the kernel config flags from 
the wiki pages and map them to Kconfig options.

Best regards,
Gerion

[1] https://wiki.gentoo.org/wiki/Docker#Kernel
[2] https://wiki.gentoo.org/wiki/Dm-crypt#Kernel_Configuration
[3] https://wiki.gentoo.org/wiki/AMDGPU#Kernel




signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Idea: User centric kernel configuration

[Gerion Entrup]

Am Freitag, 12. März 2021, 17:40:02 CET schrieb Mike Pagano:
> 
> On 3/12/21 11:34 AM, Gerion Entrup wrote:
> > Am Donnerstag, 11. März 2021, 17:41:45 CET schrieb Peter Stuge:
> >> Hi,
> >>
> >> Gerion Entrup wrote:
> >>> the Linux kernel has _a lot of_ configuration options, way too many to 
> >>> configure them by hand.
> >>
> >> I actually disagree strongly with that; I think it's important to
> >> actively decide what kernels include, and I routinely do, but of
> >> course not everyone will. I've made sure to include a kernel build
> >> when teaching systems administration courses and would again.
> >>
> >> As the kernel becomes more complex the threshold for the first
> >> configuration also rises, but it's still completely possible to learn
> >> what you need in order to successfully configure your own kernel.
> >> I guess it's on the order of a weekend project given some basic
> >> understanding of computer architecture and programming.
> > 
> > I think, we mean two different things here. I for myself also configured
> > my own kernel(s) for years. I also actively teach students to do so.
> > However, I have never looked into all 18000 configuration options. I
> > don't understand them all (by far).
> > 
> > Actually, most of the times, I do a `make localmodconfig`, click through
> > various subsystems and activate what I think that it is useful or sounds
> > nice. Then, to update a kernel, I use `make oldconfig` and answer the
> > questions as good as I can (taking the default otherwise).
> > 
> > If my kernel is not capable to do something (for example run docker
> > containers), I take a look in the Gentoo Wiki and "copy" the options
> > into my own config.
> > 
> > But for me, this is not an informed decision. Of course, I learn
> > something in this process, maybe also more than a precompiled kernel
> > user, but I have by far not created a minimal config or even begin
> > to understand all subsystems and different configuration options.
> > 
> > I'm sure that this is similar to most Gentoo users.
> > 
> > BTW, here is a project to create a minimal config (kind of "ideal"
> > config) based on a "golden run" [1]. However, I have not compared such
> > a config with my own homegrown config.
> > 
> > 
> >>> This requires a mapping between user oriented "features" and the kernel
> >>> internal configuration options.
> >>
> >> So the challenge here is that the kernel is disjoint from user space,
> >> and while the kernel API remains stable over time consumer requirements
> >> such as "docker" or "cryptsetup" will mean different things for
> >> different versions of particular user space software.
> >>
> >>
> >>> Do you think that it is useful and feasible to combine these two 
> >>> mechanisms?
> >>
> >> AFAIK there's no generic method for formal kernel requirements in user
> >> space packages and there's also no sanctioned method for quering
> >> kernel capabilities. The only thing available is /proc/config if that
> >> was enabled in the kernel build, and there are of course reasons to
> >> leave it out, and it only applies to the particular running kernel,
> >> e.g. useless for cross-compilation. There, it would be possible to
> >> read the kernel configuration file if the kernel source code is
> >> available when the userspace package is being built, but that's not
> >> guaranteed.
> >>
> >> In Gentoo, linux-info.eclass provides linux_config_exists() to do all
> >> of this, but order to become a widespread success there would have to
> >> be one method for upstreams to maintain these requirements as part of
> >> their packages, rather than forcing the burden on package maintainers
> >> to repeat the same detective task in every single distribution.
> >>
> >> I think it would be very useful to create something generic for that,
> >> but that's certainly no small task.
> >>
> >> And realistically I only see it succeeding if Linux Foundation decides
> >> to push it onto the world.
> > 
> > Sorry, I may have expressed this not clearly. I guess, we actually have
> > the same opinion here. I don't want to do this mapping automatically. I
> > don't want a "fully automatic" config or touch any (user space) ebuilds.
> > 
> > My idea is to patch Kconfig (as part of gentoo-sources)

Re: [gentoo-dev] Idea: User centric kernel configuration

[Gerion Entrup]

Am Freitag, 12. März 2021, 19:28:51 CET schrieb Mike Pagano:
> 
> On 3/12/21 1:04 PM, Gerion Entrup wrote:
> > Am Freitag, 12. März 2021, 17:40:02 CET schrieb Mike Pagano:
> >>
> >> On 3/12/21 11:34 AM, Gerion Entrup wrote:
> >>> Am Donnerstag, 11. März 2021, 17:41:45 CET schrieb Peter Stuge:
> >>>> Hi,
> >>>>
> >>>> Gerion Entrup wrote:
> >>>>> the Linux kernel has _a lot of_ configuration options, way too many to 
> >>>>> configure them by hand.
> >>>>
> >>>> I actually disagree strongly with that; I think it's important to
> >>>> actively decide what kernels include, and I routinely do, but of
> >>>> course not everyone will. I've made sure to include a kernel build
> >>>> when teaching systems administration courses and would again.
> >>>>
> >>>> As the kernel becomes more complex the threshold for the first
> >>>> configuration also rises, but it's still completely possible to learn
> >>>> what you need in order to successfully configure your own kernel.
> >>>> I guess it's on the order of a weekend project given some basic
> >>>> understanding of computer architecture and programming.
> >>>
> >>> I think, we mean two different things here. I for myself also configured
> >>> my own kernel(s) for years. I also actively teach students to do so.
> >>> However, I have never looked into all 18000 configuration options. I
> >>> don't understand them all (by far).
> >>>
> >>> Actually, most of the times, I do a `make localmodconfig`, click through
> >>> various subsystems and activate what I think that it is useful or sounds
> >>> nice. Then, to update a kernel, I use `make oldconfig` and answer the
> >>> questions as good as I can (taking the default otherwise).
> >>>
> >>> If my kernel is not capable to do something (for example run docker
> >>> containers), I take a look in the Gentoo Wiki and "copy" the options
> >>> into my own config.
> >>>
> >>> But for me, this is not an informed decision. Of course, I learn
> >>> something in this process, maybe also more than a precompiled kernel
> >>> user, but I have by far not created a minimal config or even begin
> >>> to understand all subsystems and different configuration options.
> >>>
> >>> I'm sure that this is similar to most Gentoo users.
> >>>
> >>> BTW, here is a project to create a minimal config (kind of "ideal"
> >>> config) based on a "golden run" [1]. However, I have not compared such
> >>> a config with my own homegrown config.
> >>>
> >>>
> >>>>> This requires a mapping between user oriented "features" and the kernel
> >>>>> internal configuration options.
> >>>>
> >>>> So the challenge here is that the kernel is disjoint from user space,
> >>>> and while the kernel API remains stable over time consumer requirements
> >>>> such as "docker" or "cryptsetup" will mean different things for
> >>>> different versions of particular user space software.
> >>>>
> >>>>
> >>>>> Do you think that it is useful and feasible to combine these two 
> >>>>> mechanisms?
> >>>>
> >>>> AFAIK there's no generic method for formal kernel requirements in user
> >>>> space packages and there's also no sanctioned method for quering
> >>>> kernel capabilities. The only thing available is /proc/config if that
> >>>> was enabled in the kernel build, and there are of course reasons to
> >>>> leave it out, and it only applies to the particular running kernel,
> >>>> e.g. useless for cross-compilation. There, it would be possible to
> >>>> read the kernel configuration file if the kernel source code is
> >>>> available when the userspace package is being built, but that's not
> >>>> guaranteed.
> >>>>
> >>>> In Gentoo, linux-info.eclass provides linux_config_exists() to do all
> >>>> of this, but order to become a widespread success there would have to
> >>>> be one method for upstreams to maintain these requirements as part of
> >>>> their packages, rather than forcing the burden on package maintainers
> >>

Re: [gentoo-dev] [RFC] Un-slotting LLVM

[Gerion Entrup]

Am Montag, 8. November 2021, 12:18:30 CET schrieb Michał Górny:
> WDYT?

Not a Gentoo dev but a user who develops software.
I find it pretty convenient to have multiple LLVM versions available.
I'm developing software that links against LLVM and can test it against
multiple versions pretty easy with the current setup. I can evaluate a
new LLVM version before switching my own software to it. I can even test
some old (already bit-rotted) software that needs an old LLVM version since
Gentoo provides it.

I'm pretty sure that this argument also holds for all users that want to
test their software against multiple versions of Clang (just the same as it
holds for slotted GCC).

To summarize: Slotted LLVM is not only a feature for other distribution
packages but also for users of Gentoo that develop software in compiled
languages.


> I'm hearing more
> and more reports of programs being broken through getting multiple LLVM
> versions in the link chain.

I'm actually experienced this, but slotted LLVM was a solution here not
a problem. Let me explain the link chain (an arrow means: "link against"):

my tool --> an unpackaged third party lib (bound to an old LLVM) -> LLVM 10
´-> graph-tool -> matplotlib -> Mesa (with AMD graphic card) -> LLVM 12 
(system default)

So my tool depends on a third party lib which needs LLVM 10 (without an easy 
port).
But as I also have an AMD graphic card so my Mesa is compiled with libLLVM for 
AMDGPU
which loads LLVM 12. My tool uses graph-tool which at the end loads Mesa.
As a result my LLVM versions conflicted.

My fix was to modify the Mesa ebuild to use LLVM 10 as well. This would
have been possible without slotted LLVM, but then I has to use LLVM 10
for everything in my system. I'm also not sure, if this would
be possible on any other distribution (without containerizing stuff).
(In the meantime the third party lib has updated its code to LLVM 12 so
everything is fine again.)

Best,
Gerion


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] EGO_SUM (was: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.)

[Gerion Entrup]

Am Mittwoch, 5. Juli 2023, 01:09:30 CEST schrieb Oskari Pirhonen:
> On Tue, Jul 04, 2023 at 21:56:26 +, Robin H. Johnson wrote:
> > On Tue, Jul 04, 2023 at 12:44:39PM +0200, Gerion Entrup wrote:
> > > just to be curious about the whole discussion. I did not follow in the
> > > deepest detail but what I got is:
> > > - EGO_SUM blows up the Manifest file, since every little Go module needs
> > >   to be respected. A lot of these Manifest files lead to a extremely
> > >   increased Portage tree size. EGO_SUM is just one example (though the
> > >   biggest one). Statically linked languages like Rust etc. have the same
> > >   problem.
> > > - The current solution is to prepackage all modules, put it somewhere on
> > >   a webserver and just manifest that file. This make the Portage tree
> > >   small in size again, but requires a webserver/mirror and is thus
> > >   unfriendly for overlay devs.
> > > 
> > > I'm not sure if it was mentioned before but has anyone considered hash
> > > trees / Merkle trees for the manifest file? The idea would be to hash
> > > the standard manifest file a second time if it gets too big and write
> > > down that hash as new manifest file and leave EGO_SUM as is.
> > This is out-of-tree/indirect Manifests, that I proposed here, more than
> > a year ago:
> > https://marc.info/?l=gentoo-dev=168280762310716=2
> > https://marc.info/?l=gentoo-dev=165472088822215=2
> > 
> > Developing it requires PMS work in addition to package manager
> > development, because it introduces phases.
> > 
> > - primary fetch of $SRC_URI per ebuild, including indirect Manifest
> > - primary validation of distfiles
> > - secondary fetch of $SRC_URI per indirect Manifest
> > - secondary validation of additional distfiles
> > 
> > A significantly impacted use case is "emerge -f", it now needs to run
> > downloads twice.
> > 
> 
> I'm not sure double downloading is required. Consider a flow similar to
> this:
> 
> 1. distfiles are fetched as per the ebuild
> 2. distfiles are hashed into a temporary Manifest
> 3. temporary Manifest is hashed and compared with the hashes stored in
>the in-tree Manifest for the direct Manifest

This is exactly, what I meant. A webstorage is not needed. A second
download process is also not needed. Just an additional Manifest format
is needed for ebuilds with more than n distfiles.


> A new Manifest format would be required in order to differentiate the
> current ones from an indirect one. This may require PMS changes,
> although I suspect ammending GLEP 74 may be enough since the PMS seems
> to just refer to the GLEP for a description of Manifests.
> 
> This would also either rely on a stable ordering of Manifest contents
> when generating it or having a separate file listing in the indirect
> Manifest which corresponds to the order in the direct Manifest. For the
> latter, it should also have separate entries for different package
> versions so that every single distfile for every single version of said
> package does not need to be fetched in order to build the direct
> Manifest.
> 
> I'm imagining something along these lines:
> 
> INDIRECT true
> PACKAGE category/package-version distfile1 distfile2 ... ALGO1 hash1 
> ALGO2 hash2 ...
> PACKAGE ...

Maybe it is reasonable to skip the distfile names at all (or just
provide a hash value of the concatenated file names). Then the manifest
would just contain two/three hashes (for as many distfiles as the ebuild
needs). Since these kind of indirect Manifests should be more rare than
the normal ones, a slightly longer processing time does not have much
impact I would say.



> Here `ALGO1` and `hash1` correspond to the hash of the direct Manifest
> containing the distfiles (and potentially other files if a repo does not
> have thin-manifests enabled) and their hashes in the order specified
> previously.
> 
> The indirect Manifest as described above would be large-ish for a
> package that has lots of distfiles, but likely much smaller than if each
> distfile had its set of hashes stored directly.

Without storing the filenames, the Manifest file would have the same
small size for any amount of distfiles needed.

Gerion


> Please correct me if there's some detail I've overlooked.
> 
> - Oskari
> 
> > The rest of the posts also go into the matter of duplication within
> > EGO_SUM & the indirect Manifests: limiting the growth requires some form
> > of content-addressed layout.
> > 
> > It's absolutely something we should get developed, but it's a lot of
> > work.
> > 
> > The indirect Manifests still provide a hosting challenge for overlays.
> > 
> 
> 
> 



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] EGO_SUM (was: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.)

[Gerion Entrup]

Am Dienstag, 4. Juli 2023, 09:13:30 CEST schrieb Tim Harder:
> On 2023-07-03 Mon 04:17, Florian Schmaus wrote:
> >On 30/06/2023 13.33, Eray Aslan wrote:
> >>On Fri, Jun 30, 2023 at 03:38:11AM -0600, Tim Harder wrote:
> >>>Why do we have to keep exporting the related variables that generally
> >>>cause these size issues to the environment?
> >>
> >>I really do not want to make a +1 response but this is an excellent
> >>question that we need to answer before implementing EGO_SUM.
> >
> >Could you please discuss why you make the reintroduction of EGO_SUM 
> >dependent on this question?
> 
> Just to be clear, I don't particularly care about EGO_SUM enough to gate
> its reintroduction (and don't have any leverage to do so anyway). I'm
> just tired of the circular discussions around env issues that all seem
> to avoid actual fixes, catering instead to functionality used by a
> vanishingly small subset of ebuilds in the main repo that compels a
> certain design mostly due to how portage functioned before EAPI 0.
> 
> Other than that, supporting EGO_SUM (or any other language ecosystem
> trending towards distro-unfriendly releases) is fine as long as devs are
> cognizant how the related global-scope eclass design affects everyone
> running or working on the raw repo. I hope devs continue leveraging the
> relatively recent benchmark tooling (and perhaps more future support) to
> improve their work. Along those lines, it could be nice to see sample
> benchmark data in commit messages for large, global-scope eclass work
> just to reinforce that it was taken into account.
> 
> Tim

Hi,

just to be curious about the whole discussion. I did not follow in the
deepest detail but what I got is:
- EGO_SUM blows up the Manifest file, since every little Go module needs
  to be respected. A lot of these Manifest files lead to a extremely
  increased Portage tree size. EGO_SUM is just one example (though the
  biggest one). Statically linked languages like Rust etc. have the same
  problem.
- The current solution is to prepackage all modules, put it somewhere on
  a webserver and just manifest that file. This make the Portage tree
  small in size again, but requires a webserver/mirror and is thus
  unfriendly for overlay devs.

I'm not sure if it was mentioned before but has anyone considered hash
trees / Merkle trees for the manifest file? The idea would be to hash
the standard manifest file a second time if it gets too big and write
down that hash as new manifest file and leave EGO_SUM as is.

When Portage tries to install the package, it can download all modules,
build the "normal" Manifest file like normally, but instead of directly
compare it to the Manifest in the tree it can hash it again and compare
that to the provided Manifest. With this, Portage should have more less
the same guarantees about the validity of the source code, but the
manifest file consists of just two hashes again.
What one would loose is the direct comparison of file names (they are
included in the "meta"-hash, though) or do I miss something?

Gerion


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Last rites: dev-python/debugpy

[Gerion Entrup]

Am Samstag, 10. Februar 2024, 10:36:27 CET schrieb Michał Górny:
> # Michał Górny  (2024-02-10)
> # Very fragile, keeps breaking.  Limited platform support.  Used only
> # as an optional dependency for dev-python/ipykernel.
> # Removal on 2024-03-11.  Bug #924221.
> dev-python/debugpy

This may be a run time "dependency" for several IDEs. For example,
Neovim makes use of it with this plugin [1] (I would say that nvim-dap
is the default dap plugin für Neovim. There seem to be, however, a
problem with the Gentoo installation [2]).

Gerion


[1] https://github.com/mfussenegger/nvim-dap-python
[2] https://github.com/mfussenegger/nvim-dap-python/issues/100


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] More packages up for grabs due to developer inactivity

[Gerion Entrup]

Am Mittwoch, 14. Februar 2024, 11:38:49 CET schrieb Michał Górny:
> kde-misc/bismuth

This package can be considered dead since KDE introduced a new Tiling
API [1] with Plasma 5.27, although there minimal development ongoing.

Gerion

[1] https://github.com/Bismuth-Forge/bismuth/issues/471#issuecomment-1410969462




signature.asc
Description: This is a digitally signed message part.