Re: [gentoo-portage-dev] Re: Speeding up Tree Verification

2020-06-29 Thread Zac Medico
On 6/29/20 7:15 PM, Sid Spry wrote:
> On Mon, Jun 29, 2020, at 9:13 PM, Sid Spry wrote:
>> Hello,
>>
>> I have some runnable pseudocode outlining a faster tree verification 
>> algorithm.
> 
> Ah, right. It's worth noting that even faster than this algorithm is simply 
> verifying
> a .tar.xz. Is that totally off the table? I realize it doesn't fit every 
> usecase, but it
> seems to be faster in both sync and verification time.

We've already got support for that with sync-type = webrsync. However, I
imagine sync-type = git is even better. All of the types are covered here:

https://wiki.gentoo.org/wiki/Portage_Security
-- 
Thanks,
Zac



[gentoo-portage-dev] Re: Speeding up Tree Verification

2020-06-29 Thread Sid Spry
On Mon, Jun 29, 2020, at 9:13 PM, Sid Spry wrote:
> Hello,
> 
> I have some runnable pseudocode outlining a faster tree verification 
> algorithm.

Ah, right. It's worth noting that even faster than this algorithm is simply 
verifying
a .tar.xz. Is that totally off the table? I realize it doesn't fit every 
usecase, but it
seems to be faster in both sync and verification time.



[gentoo-portage-dev] Speeding up Tree Verification

2020-06-29 Thread Sid Spry
Hello,

I have some runnable pseudocode outlining a faster tree verification algorithm.
Before I create patches I'd like to see if there is any guidance on making the
changes as unobtrusive as possible. If the radical change in algorithm is
acceptable I can work on adding the changes.

Instead of composing any kind of structured data out of the portage tree my
algorithm just lists all files and then optionally batches them out to threads.
There is a noticeable speedup by eliding the tree traversal operations which
can be seen when running the algorithm with a single thread and comparing it to
the current algorithm in gemato (which should still be discussed here?).

Some simple tests like counting all objects traversed and verified returns the
same(ish). Once it is put into portage it could be tested in detail.

There is also my partial attempt at removing the brittle interface to GnuPG
(it's not as if the current code is badly designed, just that parsing the
output of GnuPG directly is likely not the best idea).

Needs gemato, dnspython, and requests. Slightly better than random code because
I took inspiration from the existing gemato classes.

```python (veriftree.py)
#!/usr/bin/env python3
import os, sys, zlib, hashlib, tempfile, shutil, timeit
import subprocess
from typing import List
from pprint import pprint

from gemato.manifest import (
ManifestFile,
ManifestFileEntry,
)
from wkd import (
check_domain_signature,
hash_localpart,
build_web_key_uri,
stream_to_file
)
from fetchmedia import (
OpenPGPEnvironment,
setup_verification_environment
)

# 0. Top level directory (repository) contains Manifest, a PGP signature of
#blake2b and sha512 hashes of Manifest.files.gz.
# 1. Manifest.files contains hashes of each category Manifest.gz.
# 2. The category Manifest contains hashes of each package Manifest.
# 3. The package Manifest contains hashes of each package file.
#Must be aware of PMS, e.g. aux tag specifies a file in files/.

# 0. Check signature of repo Manifest.
# 1. Merge items in Manifest.files, each category Manifest, and each package
#Manifest into one big list. The path must be made absolute.
# 2. Distribute items to threads.

# To check operation compare directory tree to files appearing in all
# ManifestRecords.

class ManifestTree(object):
__slots__ = ['_directory', '_manifest_list', '_manifest_records',
'_manifest_results']

def __init__(self, directory: str):
self._directory = directory
# Tuples of (base_path, full_path).
self._manifest_list = []
self._manifest_records = []
self._manifest_results = []

def build_manifest_list(self):
for path, dirs, files in os.walk(self._directory):
#if 'glsa' in path or 'news' in path:
#if 'metadata' in path:
#continue # Skip the metadata directory for now.
# It contains a repository. Current algo barfs on Manifest
# containing only sig.

if 'Manifest.files.gz' in files:
self._manifest_list += [(path, path + '/Manifest.files.gz')]
if 'Manifest.gz' in files:
self._manifest_list += [(path, path + '/Manifest.gz')]

if path == self._directory:
continue # Skip the repo manifest. Order matters, fix 
eventually.
if 'Manifest' in files:
self._manifest_list += [(path, path + '/Manifest')]

def parse_manifests(self):
td = tempfile.TemporaryDirectory(dir='./')
for manifest in self._manifest_list:
def inner():
if manifest[1].endswith('.gz'):
name = 'Manifest.files' # Need to also handle Manifest.gz.
path = '{0}/{1}'.format(td.name, name)
subprocess.run(['sh', '-c', 'gunzip -c {0} > {1}'
.format(manifest[1], path)])
for line in open(path):
mr = ManifestRecord(line)
mr.make_absolute(manifest[0])
self._manifest_records += [mr]
else:
for line in open(manifest[1]):
if line.startswith('-'):
return # Skip the signed manifest.
mr = ManifestRecord(line)
mr.make_absolute(manifest[0])
self._manifest_records += [mr]
inner()

def verify_manifests(self):
for record in self._manifest_records:
self._manifest_results += [record.verify()]


class ManifestRecord(object):
__slots__ = ['_tag', '_abs_path', '_path', '_size', '_hashes']

def __init__(self, line: str=None):
self._tag = None
self._abs_path = None
self._path = None
self._size = None
self._hashes = []
if line:
self.from_string(line)

def 

[gentoo-dev] Last rites: */* More Py2 stuff

2020-06-29 Thread Aaron Bauman
# Aaron Bauman  (2020-06-29)
# More Py2 only stuff. Plz see -dev ML for discussions
# Remove bindings, port to Py3, etc
# Removal in 30 days
app-dicts/opendict
app-editors/editra
app-office/taskcoach
app-backup/holland
app-backup/holland-backup-example
app-backup/holland-backup-pgdump
app-backup/holland-backup-random
app-backup/holland-backup-sqlite
app-backup/holland-lib-common
app-backup/holland-lib-lvm
app-cdr/burn-cd
app-editors/leo
app-emulation/playonlinux
app-text/bibus
dev-db/SchemaSync
dev-python/squaremap
dev-util/wxglade
media-gfx/fontypython
media-gfx/fr0st
sci-chemistry/apbs
sci-chemistry/autodock
sci-chemistry/eden
sci-chemistry/p3d
sci-chemistry/pdb2pqr
sci-chemistry/pdb-tools
sci-chemistry/prodecomp
sci-chemistry/pymol-plugins-caver
sci-chemistry/pymol-plugins-dssp
sci-chemistry/pymol-plugins-promol
sci-chemistry/relax
sci-chemistry/sparky
sci-chemistry/vmd
www-apps/roundup
www-apps/viewvc
x11-misc/nts

-- 
Cheers,
Aaron


signature.asc
Description: PGP signature


Re: [gentoo-dev] RFC: packages.gentoo.org Developer Mode

2020-06-29 Thread Aaron Bauman



On June 29, 2020 6:28:30 PM EDT, Max Magorsch  wrote:
>Hi,
>
>Some time ago, the idea of integrating packages.g.o with pkgcheck and
>repology.org came up. There have already been some discussions in bug
>725702 and 725704 regarding this.
>
>The tl;dr is that the packages.g.o backend does now parse both the
>pkgcheck results as well as the repology.org data. Currently, it's
>possible to access them:
> - either by using the new GraphQL API [0]
> - or by using the 'developer mode' in the web interface
>You can enable the developer mode by clicking onto 'Switch to
>Developer Mode' in the footer [1]. Afterwards, you will see pkgcheck
>warnings in the version table as well as warnings in case a new
>version of a package is available according to repology.org (as, e.g.
>currently for x11-wm/xmonad-contrib).
>
>I think, more generally speaking, there is some information that is of
>interest for developers but might not be of interest for regular
>users. This leads to some questions:
>
>1. Would you prefer this information to be displayed in packages.g.o
>using a 'developer mode' or would you prefer a separate application
>similar to the idea of project Grumpy?
>
>2. Is there any further information apart from pkgcheck and repology
>that you would consider useful here for the development workflow?
>
>3. What else would you like to see here? For instance, I could think
>of a configurable dashboard that shows all pkgcheck warnings, new
>versions and open pull requests for packages that a developer
>maintains. Would that be useful? What else could you think of?
>
>I look forward to your opinion.
>
>-M
>
>[0] you can use https://packages.gentoo.org/api/explore/ to get started
>[1] if that's not working, that's likely a caching issue, and the
>javascript assets have to be refreshed

Hi, Max. A couple thoughts... Just let me know if they are too crazy. 

1. Could you enable the backend to ping devs/projects via email when a new 
release is available for their respective package(s)? Maybe make it optional 
for the dev/project to enroll?

2. What about a setting to allow the Python team to deprecate a particular 
interpreter then notify respective pkg owners that their ebuild needs updated?

This would possibly workaround the issues mgorny brought up regarding 
package.deprecated and noise for CI checks. Also, not sure if this is best 
implemented somewhere else first (e.g. pkgcheck) then leveraged by your work. 

Just a few quick thoughts...

-Aaron
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.



[gentoo-dev] Migration of Tyrian from Bootstrap to Foundation

2020-06-29 Thread Max Magorsch
Hi all,

I want to propose the migration of Tyrian from Bootstrap[0] to Foundation[1].

Before I continue, some background information: Tyrian is the unified
gentoo.org theme. It's currently based on Bootstrap, which in turn is
a front-end framework. Bootstrap is especially famous for its ready to
use components. That is, you can include Bootstrap and get started
without much customization. Tyrian was originally based on Bootstrap
v3. Bootstrap v4, however, turned out to become a major rewrite of the
whole project. That is, it's not just Tyrian that has to be migrated
to Bootstrap v4 but also all sites that are using Tyrian, as Bootstrap
v4 is not backwards compatible.

I've already ported Tyrian and some sites to Bootstrap v4 some time
ago[2]. However, it turns out there are still some incompatibilities.
They, in turn, are causing problems on Gentoo sites that are already
using the new Tyrian version [3].

So the tl;dr is that we will still have to invest some time here to
make everything work as before. That's why I would like to take the
opportunity to propose to rather spend the time migrating to
Foundation instead of fixing Bootstrap incompatibilities.

Foundation is another popular front-end framework, which is also
widely used (e.g. Adobe, Amazon, Cisco, Docker, Mozilla, University of
Cambridge, or The Washington Post are using it, to name just a few).
While Bootstrap is focused on providing the tools to getting started
quickly without having to customize much, Foundation is all about
customization. That is, it is providing the 'foundation' to build your
customized theme based on your needs.

While I also think the philosophy of Foundation comes closer to
Gentoo's ideas, it's mainly the technical part that makes me suggest
the migration. Foundation is made for customizing Tyrian to our needs,
while Bootstrap was a great way to get started quickly back then.

Finally, to be clear: I'm solely suggesting a migration of the
framework that is used to customize Tyrian to our needs. I'm not
suggesting any major change of the visual appearance in this mail.
While we can discuss visual changes as well on this occasion, that's
not intended to be part of this mail. The appearance can stay the same
when using Foundation.

Does anyone see any blockers here? If so, please let me know;
otherwise, I would like to start the migration, to move this forward.

-M

[0] https://getbootstrap.com/
[1] https://get.foundation/
[2] https://wiki.gentoo.org/wiki/Project:Website/Tyrian#Rollout_status
contains the rollout status
[3] Thanks to everyone who helped pointing out problems that occurred
after using the new Tyrian version



[gentoo-dev] RFC: packages.gentoo.org Developer Mode

2020-06-29 Thread Max Magorsch
Hi,

Some time ago, the idea of integrating packages.g.o with pkgcheck and
repology.org came up. There have already been some discussions in bug
725702 and 725704 regarding this.

The tl;dr is that the packages.g.o backend does now parse both the
pkgcheck results as well as the repology.org data. Currently, it's
possible to access them:
 - either by using the new GraphQL API [0]
 - or by using the 'developer mode' in the web interface
You can enable the developer mode by clicking onto 'Switch to
Developer Mode' in the footer [1]. Afterwards, you will see pkgcheck
warnings in the version table as well as warnings in case a new
version of a package is available according to repology.org (as, e.g.
currently for x11-wm/xmonad-contrib).

I think, more generally speaking, there is some information that is of
interest for developers but might not be of interest for regular
users. This leads to some questions:

1. Would you prefer this information to be displayed in packages.g.o
using a 'developer mode' or would you prefer a separate application
similar to the idea of project Grumpy?

2. Is there any further information apart from pkgcheck and repology
that you would consider useful here for the development workflow?

3. What else would you like to see here? For instance, I could think
of a configurable dashboard that shows all pkgcheck warnings, new
versions and open pull requests for packages that a developer
maintains. Would that be useful? What else could you think of?

I look forward to your opinion.

-M

[0] you can use https://packages.gentoo.org/api/explore/ to get started
[1] if that's not working, that's likely a caching issue, and the
javascript assets have to be refreshed



[gentoo-dev] packages.gentoo.org GraphQL API

2020-06-29 Thread Max Magorsch
Hi all,

Today I want to announce the new public GraphQL API for packages.g.o.
While we already had some different REST endpoints in the past, these
endpoints were neither documented nor specified. The behaviour of
these endpoints even changed over time due to the lacking
specification.

Instead of adding further REST endpoints, we are now making the
packages.g.o API available via GraphQL at:

  https://packages.gentoo.org/api/graphql/

General information about GraphQL can be found at [0]. You can
interactively explore the API using the GraphiQL explorer at [1]. The
explorer contains examples as well as documentation of the
specification.

This is the first version of the API so everyone is welcome to give it
a try and give feedback in case anything is not working as expected or
missing there.

-M

[0] https://graphql.org/
[1] https://packages.gentoo.org/api/explore/



Re: [gentoo-dev] Last rites: */* More Py2 only items

2020-06-29 Thread Sam James

> On 29 Jun 2020, at 01:35, Aaron Bauman  wrote:
> 
> # Aaron Bauman  (2020-06-28)
> # More Py2 only stuff. Plz see -dev ML for discussions
> # Remove bindings, port to Py3, etc
> 
> net-irc/irker
> 

Upstream seem to be about to release 2.19 which includes
several patches for Python 3 compatibility.

(There wasn’t much activity 24 hours ago.. maybe you inspired
something, bman!)

I’ll make a PR with these (either with the patches or release) if
they work fine.

> --
> Cheers,
> Aaron



signature.asc
Description: Message signed with OpenPGP


Re: [gentoo-dev] Re: [gentoo-dev-announce] Last rites: */* More Py2 only items

2020-06-29 Thread Aaron Bauman



On June 29, 2020 4:57:49 PM EDT, Piotr Karbowski  wrote:
>On 29/06/2020 02.35, Aaron Bauman wrote:
>> [...]
>> net-dns/maradns
>
>There's single python script that can work with Python3 (at least in
>new
>version) that does that can just use any Python version.
>
>I see that it did not got update for over a year, will take it over now
>and push
>update tomorrow, then drop your mask on this package.
>
>-- Piotr.

Piotr, perfect. Thanks for helping out!

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.



[gentoo-dev] Re: [gentoo-dev-announce] Last rites: */* More Py2 only items

2020-06-29 Thread Piotr Karbowski
On 29/06/2020 02.35, Aaron Bauman wrote:
> [...]
> net-dns/maradns

There's single python script that can work with Python3 (at least in new
version) that does that can just use any Python version.

I see that it did not got update for over a year, will take it over now and push
update tomorrow, then drop your mask on this package.

-- Piotr.



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] Last rites: */* More Py2 only items

2020-06-29 Thread Aaron Bauman
On Mon, Jun 29, 2020 at 09:48:34PM +0300, Azamat Hackimov wrote:
> > dev-python/misaka
> I've created MR for it: https://github.com/gentoo/gentoo/pull/15984
> 

Azamat, merged! Thank you.

-- 
Cheers,
Aaron


signature.asc
Description: PGP signature


Re: [gentoo-dev] Last rites: */* More Py2 only items

2020-06-29 Thread Azamat Hackimov
> dev-python/misaka
I've created MR for it: https://github.com/gentoo/gentoo/pull/15984


-- 
>From Siberia with Love!



Re: [gentoo-dev] Re: RFC: Standard build environment variables

2020-06-29 Thread Ulrich Mueller
> On Mon, 29 Jun 2020, Agostino Sarubbo wrote:

> On domenica 28 giugno 2020 14:18:23 CEST Michael Orlitzky wrote:
>> To avoid these issues, I suggest creating a list of "Gentoo environment
>> variables" in the devmanual with descriptions of how they should be used
>> and pointers to the references (for why we chose that meaning). That way
>> a user can export LD, for example, and know that it will be used how he
>> thinks it will be used.

> +1

Patches are welcome. :)


signature.asc
Description: PGP signature


[gentoo-dev] Re: RFC: Standard build environment variables

2020-06-29 Thread Agostino Sarubbo
On domenica 28 giugno 2020 14:18:23 CEST Michael Orlitzky wrote:
> To avoid these issues, I suggest creating a list of "Gentoo environment
> variables" in the devmanual with descriptions of how they should be used
> and pointers to the references (for why we chose that meaning). That way
> a user can export LD, for example, and know that it will be used how he
> thinks it will be used.

+1

Agostino






[gentoo-dev] Re: [gentoo-dev-announce] Packages up for grabs: app-doc/zeal, x11-terms/pangoterm, maybe dev-ruby/neovim-ruby-client

2020-06-29 Thread Jeroen Roovers
On Sun, 28 Jun 2020 19:26:14 +0200
Michał Górny  wrote:

> Due to the maintainer going AWOL

Surely you meant MIA.


Kind regards,
 jer



[gentoo-dev] Last rites: sys-apps/pacman

2020-06-29 Thread Michał Górny
# Michał Górny  (2020-06-29)
# Unmaintained in Gentoo.  Multiple unresolved vulnerabilities.
# Suggested replacement: sys-apps/portage (just kidding).
# Removal in 30 days.  Bug #711134.
sys-apps/pacman

-- 
Best regards,
Michał Górny



signature.asc
Description: This is a digitally signed message part


Re: [gentoo-dev] Last rites: */* More Py2 only items

2020-06-29 Thread Robin H. Johnson
On Sun, Jun 28, 2020 at 07:57:18PM -0700, Alec Warner wrote:
> On Sun, Jun 28, 2020 at 5:35 PM Aaron Bauman  wrote:
> > app-crypt/virtualsmartcard
Upstream supposedly fixed it for py3 several years ago, maybe trivial
bump?

> > net-nds/nsscache
> Infra uses this, we will have to look into bumping it.
Bump done.

> > sys-apps/x86info
Also bumped this for trivial python usage, but surprised that upstream
seems to have stopped real maintenance of it with no clear replacement
in sight.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: PGP signature


Re: [gentoo-portage-dev] [PATCH] ecompress: optimize docompress -x precompressed comparison

2020-06-29 Thread Robin H. Johnson
On Sun, Jun 28, 2020 at 12:54:56PM -0700, Zac Medico wrote:
> Use sort and comm with temporary files in order to compare lists
> of docompress -x and precompressed files, since the file lists
> can be extremely large. Also strip ${D%/} from paths in order to
> reduce length.
+1 looks much better.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: PGP signature