Re: [gentoo-portage-dev] [PATCH] Distfile Patching Support for Portage

2011-08-21 Thread Rafael Goncalves Martins 
Patch reworked, after Zac's feedback on #-portage.

Attached.

-- 
Rafael Goncalves Martins
Gentoo Linux developer
http://rafaelmartins.eng.br/
From a12f837c20d3149f077737983bdfea23c8545abb Mon Sep 17 00:00:00 2001
From: Rafael G. Martins rafaelmart...@gentoo.org
Date: Sun, 21 Aug 2011 03:42:06 -0300
Subject: [PATCH] distpatch: basic implementation of distfile patching
 support.

This patch allows Portage to call the tools from app-portage/distpatch
to reconstruct distfiles from binary deltas and validate them.
---
 cnf/make.globals   |6 ++
 man/make.conf.5|   15 
 pym/portage/const.py   |6 +-
 pym/portage/dbapi/porttree.py  |   24 ++-
 pym/portage/package/ebuild/doebuild.py |3 +
 pym/portage/package/ebuild/fetch.py|  117 +++-
 6 files changed, 150 insertions(+), 21 deletions(-)

diff --git a/cnf/make.globals b/cnf/make.globals
index 2892d50..33b93fc 100644
--- a/cnf/make.globals
+++ b/cnf/make.globals
@@ -126,6 +126,12 @@ PORTAGE_ELOG_MAILFROM=portage@localhost
 # Signing command used by repoman
 PORTAGE_GPG_SIGNING_COMMAND=gpg --sign --clearsign --yes --default-key \\${PORTAGE_GPG_KEY}\ --homedir \\${PORTAGE_GPG_DIR}\ \\${FILE}\
 
+# URL for distpatch deltas root url
+PORTAGE_DELTAS_ROOT_URL=http://soc.dev.gentoo.org/~rafaelmartins/;
+
+# distpatch command, to fetch deltas (depends on app-portage/distpatch)
+PORTAGE_DISTPATCH_COMMAND=distpatcher --db \\${DELTADB}\ --root-url \\${ROOT_URL}\ --output \\${OUTPUT_DIR}\ --input \\${INPUT_DIR}\ --distfile \${VERBOSE} \\${FILE}\
+
 #*
 #**  DO NOT EDIT THIS FILE  **
 # ***
diff --git a/man/make.conf.5 b/man/make.conf.5
index d83258c..b19b38c 100644
--- a/man/make.conf.5
+++ b/man/make.conf.5
@@ -272,6 +272,11 @@ strangely configured Samba server (oplocks off, NFS re\-export). A tool
 /usr/lib/portage/bin/clean_locks exists to help handle lock issues
 when a problem arises (normally due to a crash or disconnect).
 .TP
+.B distpatch
+Enable experimental support to Distfile Patching. It depends on
+\fBapp-portage/distpatch\fR. Partially based on GLEP 25.
+See: \fIhttp://www.gentoo.org/proj/en/infrastructure/distpatch/\fR
+.TP
 .B ebuild\-locks
 Use locks to ensure that unsandboxed ebuild phases never execute
 concurrently. Also see \fIparallel\-install\fR.
@@ -652,6 +657,16 @@ matching files are excluded when the \fBPORTAGE_COMPRESS\fR command is
 called. Regular expressions are supported and the match is performed only
 against the portion of the file name which follows the last period character.
 .TP
+\fBPORTAGE_DELTAS_ROOT_URL\fR = \fIURL\fR
+This variable contains the URL used to fetch deltas to be used by the
+\fBdistpatch\fR feature.
+.TP
+.B PORTAGE_DISTPATCH_COMMAND
+This variable contains the command used for reconstruct distfiles from old
+distfiles and deltas. It must contain the executable as well as the
+place\-holders \\${DELTADB}, \\${ROOT_URL}, \\${OUTPUT_DIR}, \\${INPUT_DIR},
+\\${VERBOSE} and \\${FILE}.
+.TP
 .B PORTAGE_ELOG_CLASSES
 .TP
 .B PORTAGE_ELOG_SYSTEM
diff --git a/pym/portage/const.py b/pym/portage/const.py
index ecaa8f1..6b211f8 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -71,6 +71,8 @@ PRELINK_BINARY   = /usr/sbin/prelink
 INVALID_ENV_FILE = /etc/spork/is/not/valid/profile.env
 REPO_NAME_FILE   = repo_name
 REPO_NAME_LOC= profiles + / + REPO_NAME_FILE
+DELTADB_FILE = delta.db
+DELTAS_DIR   = deltas
 
 PORTAGE_PACKAGE_ATOM = sys-apps/portage
 LIBC_PACKAGE_ATOM= virtual/libc
@@ -89,8 +91,8 @@ SUPPORTED_FEATURES   = frozenset([
allow-missing-manifests,
assume-digests, binpkg-logs, buildpkg, buildsyspkg, candy,
ccache, chflags, collision-protect, compress-build-logs,
-   digest, distcc, distcc-pump, distlocks, ebuild-locks, fakeroot,
-   fail-clean, fixpackages, force-mirror, getbinpkg,
+   digest, distcc, distcc-pump, distlocks, distpatch, ebuild-locks,
+   fakeroot, fail-clean, fixpackages, force-mirror, getbinpkg,
installsources, keeptemp, keepwork, fixlafiles, lmirror,
metadata-transfer, mirror, multilib-strict, news,
noauto, noclean, nodoc, noinfo, noman,
diff --git a/pym/portage/dbapi/porttree.py b/pym/portage/dbapi/porttree.py
index bf8ecd9..75a5c8f 100644
--- a/pym/portage/dbapi/porttree.py
+++ b/pym/portage/dbapi/porttree.py
@@ -15,11 +15,13 @@ portage.proxy.lazyimport.lazyimport(globals(),
 	'portage.util:ensure_dirs,shlex_split,writemsg,writemsg_level',
 	'portage.util.listdir:listdir',
 	'portage.versions:best,catpkgsplit,_pkgsplit@pkgsplit,ver_regexp',
+

Re: [gentoo-portage-dev] [PATCH] Distfile Patching Support for Portage

2011-08-21 Thread Rafael Goncalves Martins 
On Sun, Aug 21, 2011 at 3:42 PM, Rafael Goncalves Martins
rafaelmart...@gentoo.org wrote:
 Patch reworked, after Zac's feedback on #-portage.

 Attached.


Re-sending. I forgot a print statement in the code. :(

Sorry.

-- 
Rafael Goncalves Martins
Gentoo Linux developer
http://rafaelmartins.eng.br/
From 3fe23a65638160c19abe991d9db6f023818c5395 Mon Sep 17 00:00:00 2001
From: Rafael G. Martins rafaelmart...@gentoo.org
Date: Sun, 21 Aug 2011 20:32:02 -0300
Subject: [PATCH] distpatch: basic implementation of distfile patching
 support.

This patch allows Portage to call the tools from app-portage/distpatch
to reconstruct distfiles from binary deltas and validate them.
---
 cnf/make.globals   |6 ++
 man/make.conf.5|   15 
 pym/portage/const.py   |6 +-
 pym/portage/dbapi/porttree.py  |   24 ++-
 pym/portage/package/ebuild/doebuild.py |3 +
 pym/portage/package/ebuild/fetch.py|  116 +++-
 6 files changed, 149 insertions(+), 21 deletions(-)

diff --git a/cnf/make.globals b/cnf/make.globals
index 2892d50..33b93fc 100644
--- a/cnf/make.globals
+++ b/cnf/make.globals
@@ -126,6 +126,12 @@ PORTAGE_ELOG_MAILFROM=portage@localhost
 # Signing command used by repoman
 PORTAGE_GPG_SIGNING_COMMAND=gpg --sign --clearsign --yes --default-key \\${PORTAGE_GPG_KEY}\ --homedir \\${PORTAGE_GPG_DIR}\ \\${FILE}\
 
+# URL for distpatch deltas root url
+PORTAGE_DELTAS_ROOT_URL=http://soc.dev.gentoo.org/~rafaelmartins/;
+
+# distpatch command, to fetch deltas (depends on app-portage/distpatch)
+PORTAGE_DISTPATCH_COMMAND=distpatcher --db \\${DELTADB}\ --root-url \\${ROOT_URL}\ --output \\${OUTPUT_DIR}\ --input \\${INPUT_DIR}\ --distfile \${VERBOSE} \\${FILE}\
+
 #*
 #**  DO NOT EDIT THIS FILE  **
 # ***
diff --git a/man/make.conf.5 b/man/make.conf.5
index d83258c..b19b38c 100644
--- a/man/make.conf.5
+++ b/man/make.conf.5
@@ -272,6 +272,11 @@ strangely configured Samba server (oplocks off, NFS re\-export). A tool
 /usr/lib/portage/bin/clean_locks exists to help handle lock issues
 when a problem arises (normally due to a crash or disconnect).
 .TP
+.B distpatch
+Enable experimental support to Distfile Patching. It depends on
+\fBapp-portage/distpatch\fR. Partially based on GLEP 25.
+See: \fIhttp://www.gentoo.org/proj/en/infrastructure/distpatch/\fR
+.TP
 .B ebuild\-locks
 Use locks to ensure that unsandboxed ebuild phases never execute
 concurrently. Also see \fIparallel\-install\fR.
@@ -652,6 +657,16 @@ matching files are excluded when the \fBPORTAGE_COMPRESS\fR command is
 called. Regular expressions are supported and the match is performed only
 against the portion of the file name which follows the last period character.
 .TP
+\fBPORTAGE_DELTAS_ROOT_URL\fR = \fIURL\fR
+This variable contains the URL used to fetch deltas to be used by the
+\fBdistpatch\fR feature.
+.TP
+.B PORTAGE_DISTPATCH_COMMAND
+This variable contains the command used for reconstruct distfiles from old
+distfiles and deltas. It must contain the executable as well as the
+place\-holders \\${DELTADB}, \\${ROOT_URL}, \\${OUTPUT_DIR}, \\${INPUT_DIR},
+\\${VERBOSE} and \\${FILE}.
+.TP
 .B PORTAGE_ELOG_CLASSES
 .TP
 .B PORTAGE_ELOG_SYSTEM
diff --git a/pym/portage/const.py b/pym/portage/const.py
index ecaa8f1..6b211f8 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -71,6 +71,8 @@ PRELINK_BINARY   = /usr/sbin/prelink
 INVALID_ENV_FILE = /etc/spork/is/not/valid/profile.env
 REPO_NAME_FILE   = repo_name
 REPO_NAME_LOC= profiles + / + REPO_NAME_FILE
+DELTADB_FILE = delta.db
+DELTAS_DIR   = deltas
 
 PORTAGE_PACKAGE_ATOM = sys-apps/portage
 LIBC_PACKAGE_ATOM= virtual/libc
@@ -89,8 +91,8 @@ SUPPORTED_FEATURES   = frozenset([
allow-missing-manifests,
assume-digests, binpkg-logs, buildpkg, buildsyspkg, candy,
ccache, chflags, collision-protect, compress-build-logs,
-   digest, distcc, distcc-pump, distlocks, ebuild-locks, fakeroot,
-   fail-clean, fixpackages, force-mirror, getbinpkg,
+   digest, distcc, distcc-pump, distlocks, distpatch, ebuild-locks,
+   fakeroot, fail-clean, fixpackages, force-mirror, getbinpkg,
installsources, keeptemp, keepwork, fixlafiles, lmirror,
metadata-transfer, mirror, multilib-strict, news,
noauto, noclean, nodoc, noinfo, noman,
diff --git a/pym/portage/dbapi/porttree.py b/pym/portage/dbapi/porttree.py
index bf8ecd9..75a5c8f 100644
--- a/pym/portage/dbapi/porttree.py
+++ b/pym/portage/dbapi/porttree.py
@@ -15,11 +15,13 @@ portage.proxy.lazyimport.lazyimport(globals(),