Re: [arch-projects] [dbscripts] [PATCH 0/5] Be less tightly coupled with SVN

[Luke Shumaker]

On Fri, 22 Jun 2018 15:49:19 -0400,
Luke Shumaker wrote:
> On Fri, 22 Jun 2018 12:52:04 -0400,
> Eli Schwartz wrote:
> > and
> > IMHO also has a lot more code clarity than a mini awk program reading
> > from pacman's technically undocumented database format (makepkg/PKGBUILD
> > is quite well documented, I'm not sure we make any guarantees about the
> > database format except that libalpm can read it).
> 
> For what it's worth, the added AWK program is very similar to the one
> already in `sourceballs`.

Come to think of it, would you be happier with a Python program using
pyalpm to parse it?

libalpm doesn't offer a convenient way to say "please parse this .db
file", but I think that can be worked around by synthesizing a config
that has a repo pointing to it via a "Server=file:///..." setting.

-- 
Happy hacking,
~ Luke Shumaker

Re: [arch-projects] [dbscripts] [PATCH 0/5] Be less tightly coupled with SVN

[Luke Shumaker]

On Fri, 22 Jun 2018 12:52:04 -0400,
Eli Schwartz wrote:
> On 06/19/2018 12:20 AM, Luke Shumaker wrote:
> > From: Luke Shumaker 
> > 
> > This patchset is intended to clean up dbscripts' interaction with SVN.
> > Besides (IMO) improved code clarity, this should make it easier for those
> > who wish to replace SVN with something else.
> > 
> > There are two major themes here:
> >  1. Avoid asking SVN for information; to the extent possible, get all
> > information directly from the pacman repos.
> 
> Why is this a goal? Especially, because currently dbscripts will error
> out on you if you do something like create a PKGBUILD in a directory
> that doesn't match the pkgbase. This is a feature, because we don't want
> that to happen.

That should not change:
 - db-update will still do the check_pkgsvn and such that check the PKGBUILD
 - db-move, db-remove  will still  move/remove the PKGBUILD,  and will
   error if it isn't where they expect it.

Replacing SVN with something else (whether it be in Parabola, or in
dbscripts-rewrite) involves considering many interactions between
dbscripts and the VCS repo.  That's hard to do (as evidenced by issues
in the Parabola bug tracker, and by "TODO" comments in
dbscripts-rewrite).  By removing pointless interactions, we can come
up with a clean interface between dbscripts and the VCS (and I called
that interface `db-abs`)

> We want the PKGBUILDs repository to match the expected pacman repository
> action, so this information should always be sourced from the PKGBUILD.

And we want the pacman repository to match the PKGBUILDs repository,
so this information should always be sourced from the .db?

> There's a WIP rewrite here:
> https://wiki.archlinux.org/index.php/User:Bluewind/dbscripts-rewrite

Yes, I am aware of Florian's work (I did include the same URL in my
cover letter).

Though I haven't spoken with Florian, one of my big goals with this is
to make is work easier.  I believe that if this patchset is applied,
then the *only*[1] changes he will have to make are contained to the
`db-abs` file, which defines just a handful of functions that provide
a clean VCS-agnostic interface.

> Which has e.g. get_file_from_pkgrepo which cat's the PKGBUILD in a
> VCS-agnostic manner. This is the direction I think we want to take,

That's not entirely different than the direction I took it: I included
an abs_export function, which writes the files for a package (PKGBUILD
and other) to a given directory, in a VCS-agnostic manner.  I just
named it abs_export, rather than get_dir_from_pkgrepo.

> and
> IMHO also has a lot more code clarity than a mini awk program reading
> from pacman's technically undocumented database format (makepkg/PKGBUILD
> is quite well documented, I'm not sure we make any guarantees about the
> database format except that libalpm can read it).

For what it's worth, the added AWK program is very similar to the one
already in `sourceballs`.

> 
> >  2. Isolate SVN-interfacing code in to functions; making it clear what
> > interfaces the PKGBUILD tracking needs to provide.
> 
> The dbscripts-rewrite aims to do this too, but in a very different way.
> I'd like to keep to those semantics in order to reduce the effort it
> takes to merge those changes.

Like I said, one of my goals is to reduce that effort.

Unless I'm looking at the wrong git repo
(), it's already a lot of work; and
several of the interactions haven't been considered.  The check_pkgsvn
and check_splitpkgs consistency checks are simply removed and replaced
with "TODO" comments.  The bit where db-remove looks at SVN to handle
split packages (that I replaced with looking at the .db file) is
similarly replaced with a "TODO" comment.

This patchset is intended to make those tricky bits trivial.

> > This does not touch the test suite, which is still firmly SVN-dependent.

[1]: Ok, I lied; they'll have to edit `db-abs` *and* the test suite.

-- 
Happy hacking,
~ Luke Shumaker

Re: [arch-projects] [dbscripts] [PATCH 0/5] Be less tightly coupled with SVN

[Eli Schwartz via arch-projects]

On 06/19/2018 12:20 AM, Luke Shumaker wrote:
> From: Luke Shumaker 
> 
> This patchset is intended to clean up dbscripts' interaction with SVN.
> Besides (IMO) improved code clarity, this should make it easier for those
> who wish to replace SVN with something else.
> 
> There are two major themes here:
>  1. Avoid asking SVN for information; to the extent possible, get all
> information directly from the pacman repos.

Why is this a goal? Especially, because currently dbscripts will error
out on you if you do something like create a PKGBUILD in a directory
that doesn't match the pkgbase. This is a feature, because we don't want
that to happen.

We want the PKGBUILDs repository to match the expected pacman repository
action, so this information should always be sourced from the PKGBUILD.

There's a WIP rewrite here:
https://wiki.archlinux.org/index.php/User:Bluewind/dbscripts-rewrite

Which has e.g. get_file_from_pkgrepo which cat's the PKGBUILD in a
VCS-agnostic manner. This is the direction I think we want to take, and
IMHO also has a lot more code clarity than a mini awk program reading
from pacman's technically undocumented database format (makepkg/PKGBUILD
is quite well documented, I'm not sure we make any guarantees about the
database format except that libalpm can read it).

>  2. Isolate SVN-interfacing code in to functions; making it clear what
> interfaces the PKGBUILD tracking needs to provide.

The dbscripts-rewrite aims to do this too, but in a very different way.
I'd like to keep to those semantics in order to reduce the effort it
takes to merge those changes.

> This does not touch the test suite, which is still firmly SVN-dependent.
> 
> I've cc'd Florian Pritz because of his related work on migrating to git.
> https://wiki.archlinux.org/index.php/User:Bluewind/dbscripts-rewrite
> 
> Luke Shumaker (5):
>   db-move, db-remove: Don't parse PKGBUILD files
>   db-move: Clarify the flow of the preflight check
>   testing2x: Discover correct repos based on the DBEXT files, not SVN
>   Centralize all SVN access into 'abs_*' functions in a new 'db-abs'
> file
>   db-move, db-abs: Simplify the abs_move_* functions
> 
>  cron-jobs/sourceballs |   4 +-
>  db-abs| 114 ++
>  db-functions  |  61 +++---
>  db-move   |  95 +--
>  db-remove |  18 ++-
>  testing2x |  40 +++
>  6 files changed, 204 insertions(+), 128 deletions(-)
>  create mode 100644 db-abs
> 


-- 
Eli Schwartz
Bug Wrangler and Trusted User



signature.asc
Description: OpenPGP digital signature

[arch-projects] [dbscripts] [PATCH 0/5] Be less tightly coupled with SVN

[Luke Shumaker]

From: Luke Shumaker 

This patchset is intended to clean up dbscripts' interaction with SVN.
Besides (IMO) improved code clarity, this should make it easier for those
who wish to replace SVN with something else.

There are two major themes here:
 1. Avoid asking SVN for information; to the extent possible, get all
information directly from the pacman repos.
 2. Isolate SVN-interfacing code in to functions; making it clear what
interfaces the PKGBUILD tracking needs to provide.

This does not touch the test suite, which is still firmly SVN-dependent.

I've cc'd Florian Pritz because of his related work on migrating to git.
https://wiki.archlinux.org/index.php/User:Bluewind/dbscripts-rewrite

Luke Shumaker (5):
  db-move, db-remove: Don't parse PKGBUILD files
  db-move: Clarify the flow of the preflight check
  testing2x: Discover correct repos based on the DBEXT files, not SVN
  Centralize all SVN access into 'abs_*' functions in a new 'db-abs'
file
  db-move, db-abs: Simplify the abs_move_* functions

 cron-jobs/sourceballs |   4 +-
 db-abs| 114 ++
 db-functions  |  61 +++---
 db-move   |  95 +--
 db-remove |  18 ++-
 testing2x |  40 +++
 6 files changed, 204 insertions(+), 128 deletions(-)
 create mode 100644 db-abs

-- 
2.17.1