Re: [gentoo-portage-dev] Re: Speeding up Tree Verification
On 6/29/20 7:15 PM, Sid Spry wrote: > On Mon, Jun 29, 2020, at 9:13 PM, Sid Spry wrote: >> Hello, >> >> I have some runnable pseudocode outlining a faster tree verification >> algorithm. > > Ah, right. It's worth noting that even faster than this algorithm is simply > verifying > a .tar.xz. Is that totally off the table? I realize it doesn't fit every > usecase, but it > seems to be faster in both sync and verification time. We've already got support for that with sync-type = webrsync. However, I imagine sync-type = git is even better. All of the types are covered here: https://wiki.gentoo.org/wiki/Portage_Security -- Thanks, Zac
[gentoo-portage-dev] Re: Speeding up Tree Verification
On Mon, Jun 29, 2020, at 9:13 PM, Sid Spry wrote: > Hello, > > I have some runnable pseudocode outlining a faster tree verification > algorithm. Ah, right. It's worth noting that even faster than this algorithm is simply verifying a .tar.xz. Is that totally off the table? I realize it doesn't fit every usecase, but it seems to be faster in both sync and verification time.
[gentoo-portage-dev] Speeding up Tree Verification
Hello,
I have some runnable pseudocode outlining a faster tree verification algorithm.
Before I create patches I'd like to see if there is any guidance on making the
changes as unobtrusive as possible. If the radical change in algorithm is
acceptable I can work on adding the changes.
Instead of composing any kind of structured data out of the portage tree my
algorithm just lists all files and then optionally batches them out to threads.
There is a noticeable speedup by eliding the tree traversal operations which
can be seen when running the algorithm with a single thread and comparing it to
the current algorithm in gemato (which should still be discussed here?).
Some simple tests like counting all objects traversed and verified returns the
same(ish). Once it is put into portage it could be tested in detail.
There is also my partial attempt at removing the brittle interface to GnuPG
(it's not as if the current code is badly designed, just that parsing the
output of GnuPG directly is likely not the best idea).
Needs gemato, dnspython, and requests. Slightly better than random code because
I took inspiration from the existing gemato classes.
```python (veriftree.py)
#!/usr/bin/env python3
import os, sys, zlib, hashlib, tempfile, shutil, timeit
import subprocess
from typing import List
from pprint import pprint
from gemato.manifest import (
ManifestFile,
ManifestFileEntry,
)
from wkd import (
check_domain_signature,
hash_localpart,
build_web_key_uri,
stream_to_file
)
from fetchmedia import (
OpenPGPEnvironment,
setup_verification_environment
)
# 0. Top level directory (repository) contains Manifest, a PGP signature of
#blake2b and sha512 hashes of Manifest.files.gz.
# 1. Manifest.files contains hashes of each category Manifest.gz.
# 2. The category Manifest contains hashes of each package Manifest.
# 3. The package Manifest contains hashes of each package file.
#Must be aware of PMS, e.g. aux tag specifies a file in files/.
# 0. Check signature of repo Manifest.
# 1. Merge items in Manifest.files, each category Manifest, and each package
#Manifest into one big list. The path must be made absolute.
# 2. Distribute items to threads.
# To check operation compare directory tree to files appearing in all
# ManifestRecords.
class ManifestTree(object):
__slots__ = ['_directory', '_manifest_list', '_manifest_records',
'_manifest_results']
def __init__(self, directory: str):
self._directory = directory
# Tuples of (base_path, full_path).
self._manifest_list = []
self._manifest_records = []
self._manifest_results = []
def build_manifest_list(self):
for path, dirs, files in os.walk(self._directory):
#if 'glsa' in path or 'news' in path:
#if 'metadata' in path:
#continue # Skip the metadata directory for now.
# It contains a repository. Current algo barfs on Manifest
# containing only sig.
if 'Manifest.files.gz' in files:
self._manifest_list += [(path, path + '/Manifest.files.gz')]
if 'Manifest.gz' in files:
self._manifest_list += [(path, path + '/Manifest.gz')]
if path == self._directory:
continue # Skip the repo manifest. Order matters, fix
eventually.
if 'Manifest' in files:
self._manifest_list += [(path, path + '/Manifest')]
def parse_manifests(self):
td = tempfile.TemporaryDirectory(dir='./')
for manifest in self._manifest_list:
def inner():
if manifest[1].endswith('.gz'):
name = 'Manifest.files' # Need to also handle Manifest.gz.
path = '{0}/{1}'.format(td.name, name)
subprocess.run(['sh', '-c', 'gunzip -c {0} > {1}'
.format(manifest[1], path)])
for line in open(path):
mr = ManifestRecord(line)
mr.make_absolute(manifest[0])
self._manifest_records += [mr]
else:
for line in open(manifest[1]):
if line.startswith('-'):
return # Skip the signed manifest.
mr = ManifestRecord(line)
mr.make_absolute(manifest[0])
self._manifest_records += [mr]
inner()
def verify_manifests(self):
for record in self._manifest_records:
self._manifest_results += [record.verify()]
class ManifestRecord(object):
__slots__ = ['_tag', '_abs_path', '_path', '_size', '_hashes']
def __init__(self, line: str=None):
self._tag = None
self._abs_path = None
self._path = None
self._size = None
self._hashes = []
if line:
self.from_string(line)
def
[gentoo-dev] Last rites: */* More Py2 stuff
# Aaron Bauman (2020-06-29) # More Py2 only stuff. Plz see -dev ML for discussions # Remove bindings, port to Py3, etc # Removal in 30 days app-dicts/opendict app-editors/editra app-office/taskcoach app-backup/holland app-backup/holland-backup-example app-backup/holland-backup-pgdump app-backup/holland-backup-random app-backup/holland-backup-sqlite app-backup/holland-lib-common app-backup/holland-lib-lvm app-cdr/burn-cd app-editors/leo app-emulation/playonlinux app-text/bibus dev-db/SchemaSync dev-python/squaremap dev-util/wxglade media-gfx/fontypython media-gfx/fr0st sci-chemistry/apbs sci-chemistry/autodock sci-chemistry/eden sci-chemistry/p3d sci-chemistry/pdb2pqr sci-chemistry/pdb-tools sci-chemistry/prodecomp sci-chemistry/pymol-plugins-caver sci-chemistry/pymol-plugins-dssp sci-chemistry/pymol-plugins-promol sci-chemistry/relax sci-chemistry/sparky sci-chemistry/vmd www-apps/roundup www-apps/viewvc x11-misc/nts -- Cheers, Aaron signature.asc Description: PGP signature
Re: [gentoo-dev] RFC: packages.gentoo.org Developer Mode
On June 29, 2020 6:28:30 PM EDT, Max Magorsch wrote: >Hi, > >Some time ago, the idea of integrating packages.g.o with pkgcheck and >repology.org came up. There have already been some discussions in bug >725702 and 725704 regarding this. > >The tl;dr is that the packages.g.o backend does now parse both the >pkgcheck results as well as the repology.org data. Currently, it's >possible to access them: > - either by using the new GraphQL API [0] > - or by using the 'developer mode' in the web interface >You can enable the developer mode by clicking onto 'Switch to >Developer Mode' in the footer [1]. Afterwards, you will see pkgcheck >warnings in the version table as well as warnings in case a new >version of a package is available according to repology.org (as, e.g. >currently for x11-wm/xmonad-contrib). > >I think, more generally speaking, there is some information that is of >interest for developers but might not be of interest for regular >users. This leads to some questions: > >1. Would you prefer this information to be displayed in packages.g.o >using a 'developer mode' or would you prefer a separate application >similar to the idea of project Grumpy? > >2. Is there any further information apart from pkgcheck and repology >that you would consider useful here for the development workflow? > >3. What else would you like to see here? For instance, I could think >of a configurable dashboard that shows all pkgcheck warnings, new >versions and open pull requests for packages that a developer >maintains. Would that be useful? What else could you think of? > >I look forward to your opinion. > >-M > >[0] you can use https://packages.gentoo.org/api/explore/ to get started >[1] if that's not working, that's likely a caching issue, and the >javascript assets have to be refreshed Hi, Max. A couple thoughts... Just let me know if they are too crazy. 1. Could you enable the backend to ping devs/projects via email when a new release is available for their respective package(s)? Maybe make it optional for the dev/project to enroll? 2. What about a setting to allow the Python team to deprecate a particular interpreter then notify respective pkg owners that their ebuild needs updated? This would possibly workaround the issues mgorny brought up regarding package.deprecated and noise for CI checks. Also, not sure if this is best implemented somewhere else first (e.g. pkgcheck) then leveraged by your work. Just a few quick thoughts... -Aaron -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
[gentoo-dev] Migration of Tyrian from Bootstrap to Foundation
Hi all, I want to propose the migration of Tyrian from Bootstrap[0] to Foundation[1]. Before I continue, some background information: Tyrian is the unified gentoo.org theme. It's currently based on Bootstrap, which in turn is a front-end framework. Bootstrap is especially famous for its ready to use components. That is, you can include Bootstrap and get started without much customization. Tyrian was originally based on Bootstrap v3. Bootstrap v4, however, turned out to become a major rewrite of the whole project. That is, it's not just Tyrian that has to be migrated to Bootstrap v4 but also all sites that are using Tyrian, as Bootstrap v4 is not backwards compatible. I've already ported Tyrian and some sites to Bootstrap v4 some time ago[2]. However, it turns out there are still some incompatibilities. They, in turn, are causing problems on Gentoo sites that are already using the new Tyrian version [3]. So the tl;dr is that we will still have to invest some time here to make everything work as before. That's why I would like to take the opportunity to propose to rather spend the time migrating to Foundation instead of fixing Bootstrap incompatibilities. Foundation is another popular front-end framework, which is also widely used (e.g. Adobe, Amazon, Cisco, Docker, Mozilla, University of Cambridge, or The Washington Post are using it, to name just a few). While Bootstrap is focused on providing the tools to getting started quickly without having to customize much, Foundation is all about customization. That is, it is providing the 'foundation' to build your customized theme based on your needs. While I also think the philosophy of Foundation comes closer to Gentoo's ideas, it's mainly the technical part that makes me suggest the migration. Foundation is made for customizing Tyrian to our needs, while Bootstrap was a great way to get started quickly back then. Finally, to be clear: I'm solely suggesting a migration of the framework that is used to customize Tyrian to our needs. I'm not suggesting any major change of the visual appearance in this mail. While we can discuss visual changes as well on this occasion, that's not intended to be part of this mail. The appearance can stay the same when using Foundation. Does anyone see any blockers here? If so, please let me know; otherwise, I would like to start the migration, to move this forward. -M [0] https://getbootstrap.com/ [1] https://get.foundation/ [2] https://wiki.gentoo.org/wiki/Project:Website/Tyrian#Rollout_status contains the rollout status [3] Thanks to everyone who helped pointing out problems that occurred after using the new Tyrian version
[gentoo-dev] RFC: packages.gentoo.org Developer Mode
Hi, Some time ago, the idea of integrating packages.g.o with pkgcheck and repology.org came up. There have already been some discussions in bug 725702 and 725704 regarding this. The tl;dr is that the packages.g.o backend does now parse both the pkgcheck results as well as the repology.org data. Currently, it's possible to access them: - either by using the new GraphQL API [0] - or by using the 'developer mode' in the web interface You can enable the developer mode by clicking onto 'Switch to Developer Mode' in the footer [1]. Afterwards, you will see pkgcheck warnings in the version table as well as warnings in case a new version of a package is available according to repology.org (as, e.g. currently for x11-wm/xmonad-contrib). I think, more generally speaking, there is some information that is of interest for developers but might not be of interest for regular users. This leads to some questions: 1. Would you prefer this information to be displayed in packages.g.o using a 'developer mode' or would you prefer a separate application similar to the idea of project Grumpy? 2. Is there any further information apart from pkgcheck and repology that you would consider useful here for the development workflow? 3. What else would you like to see here? For instance, I could think of a configurable dashboard that shows all pkgcheck warnings, new versions and open pull requests for packages that a developer maintains. Would that be useful? What else could you think of? I look forward to your opinion. -M [0] you can use https://packages.gentoo.org/api/explore/ to get started [1] if that's not working, that's likely a caching issue, and the javascript assets have to be refreshed
[gentoo-dev] packages.gentoo.org GraphQL API
Hi all, Today I want to announce the new public GraphQL API for packages.g.o. While we already had some different REST endpoints in the past, these endpoints were neither documented nor specified. The behaviour of these endpoints even changed over time due to the lacking specification. Instead of adding further REST endpoints, we are now making the packages.g.o API available via GraphQL at: https://packages.gentoo.org/api/graphql/ General information about GraphQL can be found at [0]. You can interactively explore the API using the GraphiQL explorer at [1]. The explorer contains examples as well as documentation of the specification. This is the first version of the API so everyone is welcome to give it a try and give feedback in case anything is not working as expected or missing there. -M [0] https://graphql.org/ [1] https://packages.gentoo.org/api/explore/
Re: [gentoo-dev] Last rites: */* More Py2 only items
> On 29 Jun 2020, at 01:35, Aaron Bauman wrote: > > # Aaron Bauman (2020-06-28) > # More Py2 only stuff. Plz see -dev ML for discussions > # Remove bindings, port to Py3, etc > > net-irc/irker > Upstream seem to be about to release 2.19 which includes several patches for Python 3 compatibility. (There wasn’t much activity 24 hours ago.. maybe you inspired something, bman!) I’ll make a PR with these (either with the patches or release) if they work fine. > -- > Cheers, > Aaron signature.asc Description: Message signed with OpenPGP
Re: [gentoo-dev] Re: [gentoo-dev-announce] Last rites: */* More Py2 only items
On June 29, 2020 4:57:49 PM EDT, Piotr Karbowski wrote: >On 29/06/2020 02.35, Aaron Bauman wrote: >> [...] >> net-dns/maradns > >There's single python script that can work with Python3 (at least in >new >version) that does that can just use any Python version. > >I see that it did not got update for over a year, will take it over now >and push >update tomorrow, then drop your mask on this package. > >-- Piotr. Piotr, perfect. Thanks for helping out! -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
[gentoo-dev] Re: [gentoo-dev-announce] Last rites: */* More Py2 only items
On 29/06/2020 02.35, Aaron Bauman wrote: > [...] > net-dns/maradns There's single python script that can work with Python3 (at least in new version) that does that can just use any Python version. I see that it did not got update for over a year, will take it over now and push update tomorrow, then drop your mask on this package. -- Piotr. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Last rites: */* More Py2 only items
On Mon, Jun 29, 2020 at 09:48:34PM +0300, Azamat Hackimov wrote: > > dev-python/misaka > I've created MR for it: https://github.com/gentoo/gentoo/pull/15984 > Azamat, merged! Thank you. -- Cheers, Aaron signature.asc Description: PGP signature
Re: [gentoo-dev] Last rites: */* More Py2 only items
> dev-python/misaka I've created MR for it: https://github.com/gentoo/gentoo/pull/15984 -- >From Siberia with Love!
Re: [gentoo-dev] Re: RFC: Standard build environment variables
> On Mon, 29 Jun 2020, Agostino Sarubbo wrote: > On domenica 28 giugno 2020 14:18:23 CEST Michael Orlitzky wrote: >> To avoid these issues, I suggest creating a list of "Gentoo environment >> variables" in the devmanual with descriptions of how they should be used >> and pointers to the references (for why we chose that meaning). That way >> a user can export LD, for example, and know that it will be used how he >> thinks it will be used. > +1 Patches are welcome. :) signature.asc Description: PGP signature
[gentoo-dev] Re: RFC: Standard build environment variables
On domenica 28 giugno 2020 14:18:23 CEST Michael Orlitzky wrote: > To avoid these issues, I suggest creating a list of "Gentoo environment > variables" in the devmanual with descriptions of how they should be used > and pointers to the references (for why we chose that meaning). That way > a user can export LD, for example, and know that it will be used how he > thinks it will be used. +1 Agostino
[gentoo-dev] Re: [gentoo-dev-announce] Packages up for grabs: app-doc/zeal, x11-terms/pangoterm, maybe dev-ruby/neovim-ruby-client
On Sun, 28 Jun 2020 19:26:14 +0200 Michał Górny wrote: > Due to the maintainer going AWOL Surely you meant MIA. Kind regards, jer
[gentoo-dev] Last rites: sys-apps/pacman
# Michał Górny (2020-06-29) # Unmaintained in Gentoo. Multiple unresolved vulnerabilities. # Suggested replacement: sys-apps/portage (just kidding). # Removal in 30 days. Bug #711134. sys-apps/pacman -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Last rites: */* More Py2 only items
On Sun, Jun 28, 2020 at 07:57:18PM -0700, Alec Warner wrote: > On Sun, Jun 28, 2020 at 5:35 PM Aaron Bauman wrote: > > app-crypt/virtualsmartcard Upstream supposedly fixed it for py3 several years ago, maybe trivial bump? > > net-nds/nsscache > Infra uses this, we will have to look into bumping it. Bump done. > > sys-apps/x86info Also bumped this for trivial python usage, but surprised that upstream seems to have stopped real maintenance of it with no clear replacement in sight. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] ecompress: optimize docompress -x precompressed comparison
On Sun, Jun 28, 2020 at 12:54:56PM -0700, Zac Medico wrote:
> Use sort and comm with temporary files in order to compare lists
> of docompress -x and precompressed files, since the file lists
> can be extremely large. Also strip ${D%/} from paths in order to
> reduce length.
+1 looks much better.
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
signature.asc
Description: PGP signature
