[gentoo-dev] [RFC] New go.gentoo.org url shortener

2020-12-08 Thread Max Magorsch
Hi all,

the tl;dr is that I've set up a new url shortener which is available
on https://go.gentoo.org/ currently. It can generate short links like
'go.gentoo.org/XXX' or you can create custom persistent links like
'go.gentoo.org/arzano/tyrian' or 'go.gentoo.org/infra/hashicorp'.
Basically, I'm looking for some quick feedback whether you consider
this as useful or don't think you would use it much, to decide whether
it's worth offering this service.

As mentioned the url-shortener is deployed on https://go.gentoo.org
currently. The source code is available at [0]. It's only accessible
by Gentoo developers using Gentoo SSO for the authentication. You can
log in using your nickname and your ldap password. Basically two use
cases are supported:

1) Just paste a link and get an automatically shortened url like
'go.gentoo.org/XXX'. This is pretty much the normal url shortener

2) You can paste a link and create a custom persistent link using a
prefix like 'go.gentoo.org/$prefix/$custom'. Allowed prefixes are the
nickname of the developer that is logged in or a project the developer
is part of. Thus I for one could create links like
'go.gentoo.org/arzano/tyrian' or 'go.gentoo.org/infra/hashicorp'.
However, I would e.g. not be able to create
'go.gentoo.org/qa/policy-guide' as I am not part of the qa project.
Only members of the QA team could create that link. That's also the
reason I've been coming up with this, as this way every developer /
every project would be able to create custom persistent, nice / short

However, please note that this is just a first beta version yet and
it's not meant to be production-ready yet. Please feel free to test it
though. In general, I'm looking for some feedback on whether you would
be interested in the two functionalities to get a feeling whether it's
worth spending time on this / offering this service at all.
Also I would be especially interested whether you consider the second
functionality as useful. Because if we just want the first
functionality, we can simplify this and just use any of the
url-shorteners that are already out there, instead of using the
solution I have built.


[0] https://gitweb.gentoo.org/sites/go-gentoo.git/

[gentoo-dev] New customization options available on packages.g.o

2020-10-06 Thread Max Magorsch
Hi all,

the tl;dr is that new customization options are available on
packages.gentoo.org. On the right side of the navigation bar you will
find a new link to the preferences pages. On these pages it's possible
to customize different parts of packages.g.o to your needs.

For example, it's possible to customize the keywords that are shown or
the classes of pkgcheck warnings that are displayed. Furthermore it's
also possible to include all packages, QA reports, pull requests and
bugs of projects a maintainer is part of on the maintainer page. This
way, it's for instance possible to view all bugs of packages you are
maintaining as well as bugs of packages that are maintained by
projects you are part of.

Further customization options are available on the preferences pages.
Feel free to let me know if you are missing anything. Apart from that:
Happy customizing.


[gentoo-dev] RFC: pgo - a command line interface for packages.g.o

2020-09-01 Thread Max Magorsch
Hi all,

The tl;dr is that I'm glad to announce pgo[0] - a command-line
interface for packages.gentoo.org and I'm looking for your feedback

Some more information: Some time ago I announced the new packages.g.o
GraphQL API. Now, pgo is using this API to display information about
packages like versions, metadata, dependencies, QA reports, pull
requests, bugs and the changelog on the command line. It will also be
able to display information for maintainers, as a list of packages
they are maintaining, a list of outdated packages, or bugs related to
the packages they are maintaining. Further functionality might be
added over time.

We do already have a bunch of tools which do display package
information on the command line. Furthermore, these tools do have the
advantage that they work locally. However, the reason I still came up
with pgo is that pgo can now display information (such as bugs, pull
requests, qa-reports or information for maintainers) that other tools
cannot display and bundle these information in one central tool.
That's why I'm looking forward to your feedback here: Do you think
this is something you would be interested in? Do you have any features
you like to see included in this case?

pgo is packaged as app-portage/pgo (you might prefer the live version
as it might still change quickly currently) so that you can give it a
try. You might for instance try

  $ pgo xmona

to show the package information for x11-wm/xmonad. Or try

  $ pgo -s xmona

to search for packages. Or e.g.

 $ pgo -bp grub

to show all bugs and pull requests for sys-boot/grub. However, to be
clear: Currently, it's still in a kind of "Proof of concept" stage. So
don't expect it to be fully useful right now.

So please let me know if you think it's worth spending more time on
this. I'm looking forward to your feedback.


[0] https://packages.gentoo.org/packages/app-portage/pgo

Re: [gentoo-dev] Improving warnings on packages.g.o

2020-08-27 Thread Max Magorsch
On Thu, Aug 27, 2020 at 12:10 PM Alexis Ballier  wrote:
> Any plans on using the remoteid from metadata.xml ?
> It's likely to be much more accurate since people have been filling
> those manually for some time now ;)

I think we could use the remoteid in future to improve the repology
warnings. However, I am not fully convinced that we can completely
replace the repology check using the remoteid, as a simple grep shows,
that currently 9873 packages contain at least one remote id. That's
just half of our packages in the tree. So I could imagine that we do
use the remoteid whenever it is available and otherwise fall back on
the repology data. However, for this to work we would need to things

1) Get the latest version available given the remoteid
2) Compare the latest version available with our latest version

It's especially the second part that I think is not done that quickly.
However, if anyone is interested in coming up with some logic / code
for this, any contributions are highly welcome.

Re: [gentoo-dev] Improving warnings on packages.g.o

2020-08-27 Thread Max Magorsch
On Thu, Aug 27, 2020 at 12:00 PM Nils Freydank  wrote:
> Hi,
> thanks for the work around p.g.o! I'd directly like to use your opt out for a
> package I maintain in proxied mode:
> The following changes since commit b1e0b4c9ec76049631cbb0a2d3798adbce2075ac:
>   Ignore net-misc/dropbox::void_x86_64 for repology (2020-08-26 19:14:45 
> +0200)
> are available in the Git repository at:
>   https://git.holgersson.xyz/gentoo-related/soko-metadata ignore-mktorrent
> for you to fetch changes up to dd823eaaf3f4893b24130360c7ec1c7f22f8b39b:
>   Ignore net-p2p/mktorrent for repology (2020-08-27 11:45:34 +0200)

Great, thank you!

I've just added net-p2p/mktorrent to the ignore list and the warning
should have gone on packages.g.o now.

[gentoo-dev] Improving warnings on packages.g.o

2020-08-26 Thread Max Magorsch
Hi all,

Good news regarding packages.g.o!

While the new packages.g.o version went into production some time ago,
this also led to some false warnings about outdated package versions.
This is because we currently take information about outdated package
versions from repology.org, which are not always accurate.

To avoid these false warnings it's now possible to filter the repology
checks. To do so, there is a new git repository called
soko-metadata[0]. The repository contains a folder called repology,
which in turn contains three files:

1) ignored-repositories: You can add repositories that should be
globally ignored for the repology checks for *all* packages. E.g.
Windows only repositories as appget might be disabled globally here.

2) ignored-categories: You can disable the repology check for whole
categories here. The check will be disabled for all packages in this

3) ignored-packages: You can either disable the repology check for a
single package here (e.g. sys-auth/pambase), or disable a repository
for the repology check for a single package here using atom::repo
(e.g.: net-misc/dropbox::void_x86_64). The later option is especially
useful if another distribution creates a non-standard version for a
single package and all versions of other distributions are marked as
outdated because of this (as in the case of dropbox and void linux).

The soko-metadata repository is writable by all devs, so that all
package maintainers are welcome to contribute to reduce the number of
false warnings.


[0] https://gitweb.gentoo.org/sites/soko-metadata.git/

[gentoo-dev] RFC: packages.g.o: new features available

2020-07-07 Thread Max Magorsch
Hi all,

I am glad to announce further progress at packages.gentoo.org (pgo in
the following). Compared to previous work during the last months,
which mostly addressed the back end, the new changes are rather
comprehensive. That's why I am looking forward to feedback here.

So the tl;dr is that the new pgo version now displays:
  - dependencies
  - reverse dependencies
  - pkgcheck results
  - repology.org data
  - github pull requests
  - stabilization bugs
  - keywording bugs
  - security bugs
  - general bugs
for each package.

Additionally, there are new sites for all package maintainers, that is:
  - Gentoo Projects (e.g. pyt...@gentoo.org)
  - Gentoo Developers  (e.g. la...@gentoo.org
  - Proxied Maintainers (e.g. la...@the-cow.de)

The maintainer's sites contain:
  - all packages of the maintainer
  - all outdated packages (according to repology)
  - all related pull requests
  - all related bugs (general, keywording and stabilization)
  - all security bugs
  - the changelog of all maintained packages
You will find the new sites under the 'Maintainers' tab.

The overall appearance of the site has also slightly changed to
display all of the new information.

Currently, the new version is deployed to:


Everyone is welcome to take a look around and tell me whether you
consider the new information as useful for your workflow and/or what
you would still change.

I'm looking forward to your feedback.


Re: [gentoo-dev] RFC: packages.gentoo.org Developer Mode

2020-06-30 Thread Max Magorsch
Hi Aaron,

Thanks for your suggestion.

On Tue, Jun 30, 2020 at 1:16 AM Aaron Bauman  wrote:
> Hi, Max. A couple thoughts... Just let me know if they are too crazy.
> 1. Could you enable the backend to ping devs/projects via email when a new 
> release is available for their respective package(s)? Maybe make it optional 
> for the dev/project to enroll?

I could imagine packages.g.o to provide different feeds (e.g. about
new releases of package(s) on a per maintainer/project basis).
Following this idea:

1. Everyone might directly subscribe to the packages.g.o feeds he is
interested in, to get notified

2. We might additionally create a sidecar application which consumes
the feeds and notifies developers/projects. This might be configurable
so that developers and projects can choose the warnings they would
like to receive.

> 2. What about a setting to allow the Python team to deprecate a particular 
> interpreter then notify respective pkg owners that their ebuild needs updated?
> This would possibly workaround the issues mgorny brought up regarding 
> package.deprecated and noise for CI checks. Also, not sure if this is best 
> implemented somewhere else first (e.g. pkgcheck) then leveraged by your work.

Same goes for your second idea. The sidecar application might also
handle notifications like that in this scenario.

By splitting this into two applications, packages.g.o would continue
to focus on providing the package data while we might get a second
application which handles all of the notifications.

What do you think?


[gentoo-dev] Migration of Tyrian from Bootstrap to Foundation

2020-06-29 Thread Max Magorsch
Hi all,

I want to propose the migration of Tyrian from Bootstrap[0] to Foundation[1].

Before I continue, some background information: Tyrian is the unified
gentoo.org theme. It's currently based on Bootstrap, which in turn is
a front-end framework. Bootstrap is especially famous for its ready to
use components. That is, you can include Bootstrap and get started
without much customization. Tyrian was originally based on Bootstrap
v3. Bootstrap v4, however, turned out to become a major rewrite of the
whole project. That is, it's not just Tyrian that has to be migrated
to Bootstrap v4 but also all sites that are using Tyrian, as Bootstrap
v4 is not backwards compatible.

I've already ported Tyrian and some sites to Bootstrap v4 some time
ago[2]. However, it turns out there are still some incompatibilities.
They, in turn, are causing problems on Gentoo sites that are already
using the new Tyrian version [3].

So the tl;dr is that we will still have to invest some time here to
make everything work as before. That's why I would like to take the
opportunity to propose to rather spend the time migrating to
Foundation instead of fixing Bootstrap incompatibilities.

Foundation is another popular front-end framework, which is also
widely used (e.g. Adobe, Amazon, Cisco, Docker, Mozilla, University of
Cambridge, or The Washington Post are using it, to name just a few).
While Bootstrap is focused on providing the tools to getting started
quickly without having to customize much, Foundation is all about
customization. That is, it is providing the 'foundation' to build your
customized theme based on your needs.

While I also think the philosophy of Foundation comes closer to
Gentoo's ideas, it's mainly the technical part that makes me suggest
the migration. Foundation is made for customizing Tyrian to our needs,
while Bootstrap was a great way to get started quickly back then.

Finally, to be clear: I'm solely suggesting a migration of the
framework that is used to customize Tyrian to our needs. I'm not
suggesting any major change of the visual appearance in this mail.
While we can discuss visual changes as well on this occasion, that's
not intended to be part of this mail. The appearance can stay the same
when using Foundation.

Does anyone see any blockers here? If so, please let me know;
otherwise, I would like to start the migration, to move this forward.


[0] https://getbootstrap.com/
[1] https://get.foundation/
[2] https://wiki.gentoo.org/wiki/Project:Website/Tyrian#Rollout_status
contains the rollout status
[3] Thanks to everyone who helped pointing out problems that occurred
after using the new Tyrian version

[gentoo-dev] RFC: packages.gentoo.org Developer Mode

2020-06-29 Thread Max Magorsch

Some time ago, the idea of integrating packages.g.o with pkgcheck and
repology.org came up. There have already been some discussions in bug
725702 and 725704 regarding this.

The tl;dr is that the packages.g.o backend does now parse both the
pkgcheck results as well as the repology.org data. Currently, it's
possible to access them:
 - either by using the new GraphQL API [0]
 - or by using the 'developer mode' in the web interface
You can enable the developer mode by clicking onto 'Switch to
Developer Mode' in the footer [1]. Afterwards, you will see pkgcheck
warnings in the version table as well as warnings in case a new
version of a package is available according to repology.org (as, e.g.
currently for x11-wm/xmonad-contrib).

I think, more generally speaking, there is some information that is of
interest for developers but might not be of interest for regular
users. This leads to some questions:

1. Would you prefer this information to be displayed in packages.g.o
using a 'developer mode' or would you prefer a separate application
similar to the idea of project Grumpy?

2. Is there any further information apart from pkgcheck and repology
that you would consider useful here for the development workflow?

3. What else would you like to see here? For instance, I could think
of a configurable dashboard that shows all pkgcheck warnings, new
versions and open pull requests for packages that a developer
maintains. Would that be useful? What else could you think of?

I look forward to your opinion.


[0] you can use https://packages.gentoo.org/api/explore/ to get started
[1] if that's not working, that's likely a caching issue, and the
javascript assets have to be refreshed

[gentoo-dev] packages.gentoo.org GraphQL API

2020-06-29 Thread Max Magorsch
Hi all,

Today I want to announce the new public GraphQL API for packages.g.o.
While we already had some different REST endpoints in the past, these
endpoints were neither documented nor specified. The behaviour of
these endpoints even changed over time due to the lacking

Instead of adding further REST endpoints, we are now making the
packages.g.o API available via GraphQL at:


General information about GraphQL can be found at [0]. You can
interactively explore the API using the GraphiQL explorer at [1]. The
explorer contains examples as well as documentation of the

This is the first version of the API so everyone is welcome to give it
a try and give feedback in case anything is not working as expected or
missing there.


[0] https://graphql.org/
[1] https://packages.gentoo.org/api/explore/