Re: [pacman-dev] Versioned packages

[Allan McRae]

On 10/09/16 08:41, Sergey Petrenko via pacman-dev wrote:
> Here is my attempt to solve seven years old infamous problem:
> https://bugs.archlinux.org/task/16702
> 
> Patch won't solve problem out of the box, a small changes in kernel PKGBUILD
> will be required, but only concerning install part.
> 
> Idea behind patch is pretty simple:
> 1) Configure list of packages and number of old versions pacman should try 
> to preserve.
> 2) When upgading to new version, keep old in place, if it has no file 
> conflicts with new one, and mark it as `archived`, remove oldest `archived` 
> version instead.
> 
> Most of time pacman treats `archived` packages as if they aren't installed.
> For now it won't check package conflicts and dependencies, only file 
> conflicts 
> with newer versions. It's only an outline of full solution, proof of concept 
> to illustrate the idea.
> 
> I'd like to hear opinion of community whether this problem should be solved 
> at all, or is it more like a feature of ArchLinux, and if it should, whether
> such approach suits ArchLinux's philosophy.
> 

How is this better than having a package file sitting in the cache?

The "kernel problem" in Arch is not because it is not possible to have
multiple kernel packages available.  Other distributions provide endless
amounts of kernels (e.g. Manjaro).

I don't see anything that needs done on the package manager end for this.

Allan

[pacman-dev] [PATCH 2/2] Ability to keep several old versions of package when they have no file conflicts with new version.

[Sergey Petrenko via pacman-dev]

I had gave a big thought to the question, who should decide whether to try to
archive package. 

At first glance it looks like maintainer's responsibility. She/he/it creates a 
package without file conflicts intending possibility for multiple versions to 
be installed on a system. Yet this means boolean flag on packages in sync 
repository. Such flag will look really really ugly (at least to me).

Other option is no option at all. Pacman always checks file conflicts between
old and new versions and can try to archive any packages that has none. I guess
such approach wouldn't change a bit neither in speed of pacman operations, nor 
in
file system of users with current state of sync databases. It looks like every
package has at least one common file between versions. Yet at some point such
packages may appear and why would user want to keep multiple versions of package
neither she/he/it, nor maintainer intended to be `archived`?

And thus solution I have ended with - new config option. `MaxArchived` specifies
window of versions to `archive`, and `ArchivePkg` - patterns of packages' names.

In this patch `archived` packages are handled only in update and remove 
operations.
Of course additional changes are required to display `archived` packages in 
queries,
possibly package-level conflicts checking for `archived` packages is also 
required.

---
 lib/libalpm/add.c  | 177 +++--
 lib/libalpm/alpm.h |  28 +++-
 lib/libalpm/be_local.c |  62 ++---
 lib/libalpm/conflict.c |  25 ++-
 lib/libalpm/db.c   |  22 ++
 lib/libalpm/db.h   |   3 +
 lib/libalpm/filelist.c |  54 +++
 lib/libalpm/filelist.h |   5 ++
 lib/libalpm/handle.c   |  38 +++
 lib/libalpm/handle.h   |   2 +
 lib/libalpm/package.c  |  13 
 lib/libalpm/package.h  |   2 +
 lib/libalpm/pkghash.c  |  48 ++
 lib/libalpm/pkghash.h  |   8 +++
 lib/libalpm/remove.c   |  16 +
 src/pacman/callback.c  |  21 ++
 src/pacman/conf.c  |  20 ++
 src/pacman/conf.h  |   2 +
 18 files changed, 529 insertions(+), 17 deletions(-)

diff --git a/lib/libalpm/add.c b/lib/libalpm/add.c
index d132e52..eb035d7 100644
--- a/lib/libalpm/add.c
+++ b/lib/libalpm/add.c
@@ -46,6 +46,7 @@
 #include "db.h"
 #include "remove.h"
 #include "handle.h"
+#include "filelist.h"
 
 /** Add a package to the transaction. */
 int SYMEXPORT alpm_add_pkg(alpm_handle_t *handle, alpm_pkg_t *pkg)
@@ -393,6 +394,173 @@ static int extract_single_file(alpm_handle_t *handle, 
struct archive *archive,
return errors;
 }
 
+static int should_archive_oldpkg(alpm_pkg_t *oldpkg, alpm_package_operation_t 
operation)
+{
+   return oldpkg->reason == ALPM_PKG_REASON_EXPLICIT &&
+   oldpkg->handle->max_archived > 0 &&
+   operation != ALPM_PACKAGE_REINSTALL &&
+   alpm_list_find(oldpkg->handle->archivepkg, 
oldpkg->name, _alpm_fnmatch) != NULL;
+}
+
+static int should_remove_archivable_oldpkg(alpm_pkg_t *oldpkg, alpm_pkg_t 
*newpkg,
+   alpm_package_operation_t operation, int *files_overlap)
+{
+   if (operation == ALPM_PACKAGE_DOWNGRADE) {
+   return 1;
+   }
+   *files_overlap = _alpm_filelist_overlap(>files, >files);
+   return *files_overlap;
+}
+
+static alpm_errno_t ask_to_remove_archived_pkgs(alpm_handle_t *handle,
+   alpm_pkg_t *newpkg, alpm_list_t *archived)
+{
+   alpm_errno_t ret = (alpm_errno_t)0;
+
+   if (archived) {
+   alpm_question_remove_from_archive_t question = {
+   .type = ALPM_QUESTION_REMOVE_FROM_ARCHIVE,
+   .remove = 0,
+   .pkgs = archived
+   };
+
+   QUESTION(handle, );
+   if(question.remove) {
+   alpm_list_t *itr;
+   for (itr = archived; itr; itr = 
alpm_list_next(archived)) {
+   alpm_pkg_t *archived_pkg = itr->data;
+
+   _alpm_log(handle, ALPM_LOG_DEBUG, _("removing 
archived %s-%s\n"),
+   archived_pkg->name, 
archived_pkg->version);
+
+   if(_alpm_remove_single_package(handle, 
archived_pkg, newpkg, 0, 0) == -1) {
+   ret = ALPM_ERR_TRANS_ABORT;
+   goto cleanup;
+   }
+   }
+   } else {
+   ret = ALPM_ERR_TRANS_ABORT;
+   goto cleanup;
+   }
+   }
+
+cleanup:
+   alpm_list_free(archived);
+   return ret;
+}
+
+static alpm_list_t *get_old_and_conficting_archived_pkgs(alpm_handle_t *handle,
+   alpm_pkg_t *newpkg, int oldpkg_files_overlap)
+{
+   alpm_list_t *archived = _alpm_db_get_archived_pkgs(handle->db_local, 
newpkg->name);
+   

[pacman-dev] [PATCH 1/2] Tests for expected behavior of archiving mechanism. Edits to testing framework required for handling multiple versions of package.

[Sergey Petrenko via pacman-dev]

Unfortunately, some changes are required to testing framework to handle
multiple versions of package.
New tests should make expected behavior of `archived` packages pretty clear.

---
 test/pacman/pmdb.py  | 51 +++-
 test/pacman/pmpkg.py |  2 ++
 test/pacman/pmrule.py| 17 ++
 test/pacman/tests/TESTS  |  5 
 test/pacman/tests/archived001.py | 22 +
 test/pacman/tests/archived002.py | 31 
 test/pacman/tests/archived003.py | 37 +
 test/pacman/tests/archived004.py | 31 
 test/pacman/tests/archived005.py | 29 +++
 test/pacman/tests/archived006.py | 39 ++
 10 files changed, 253 insertions(+), 11 deletions(-)
 create mode 100644 test/pacman/tests/archived001.py
 create mode 100644 test/pacman/tests/archived002.py
 create mode 100644 test/pacman/tests/archived003.py
 create mode 100644 test/pacman/tests/archived004.py
 create mode 100644 test/pacman/tests/archived005.py
 create mode 100644 test/pacman/tests/archived006.py

diff --git a/test/pacman/pmdb.py b/test/pacman/pmdb.py
index 1f20506..db2f270 100644
--- a/test/pacman/pmdb.py
+++ b/test/pacman/pmdb.py
@@ -60,6 +60,7 @@ def __init__(self, treename, root):
 self.is_local = True
 self.read_dircache = None
 self.read_pkgcache = {}
+self.archived = {}
 else:
 self.dbdir = None
 self.dbfile = os.path.join(root, util.PM_SYNCDBPATH, treename + 
".db")
@@ -79,35 +80,53 @@ def getpkg(self, name):
 if name == pkg.name:
 return pkg
 
-def db_read(self, name):
+def db_read(self, name, archived=False):
 if not self.dbdir or not os.path.isdir(self.dbdir):
 return None
 
-dbentry = None
+if not archived and name in self.read_pkgcache:
+return self.read_pkgcache[name]
+if archived and name in self.archived:
+return self.archived[name]
+
 if self.read_dircache is None:
 self.read_dircache = os.listdir(self.dbdir)
+
+candidates = []
 for entry in self.read_dircache:
 if entry == "ALPM_DB_VERSION":
 continue
 [pkgname, pkgver, pkgrel] = entry.rsplit("-", 2)
 if pkgname == name:
-dbentry = entry
-break
-if dbentry is None:
+candidates.append(entry)
+if len(candidates) == 0:
 return None
 
-if pkgname in self.read_pkgcache:
-return self.read_pkgcache[pkgname]
+for candidate in candidates:
+self.pkg_load(candidate)
+
+if name not in self.archived:
+self.archived[name] = []
+
+if not archived:
+if name in self.read_pkgcache:
+return self.read_pkgcache[name]
+else:
+return self.archived[name]
+
+return None
 
+def pkg_load(self, dbentry):
+
+[pkgname, pkgver, pkgrel] = dbentry.rsplit("-", 2)
 pkg = pmpkg.pmpkg(pkgname, pkgver + "-" + pkgrel)
-self.read_pkgcache[pkgname] = pkg
 
 path = os.path.join(self.dbdir, dbentry)
 # desc
 filename = os.path.join(path, "desc")
 if not os.path.isfile(filename):
 tap.bail("invalid db entry found (desc missing) for pkg " + 
pkgname)
-return None
+return
 fd = open(filename, "r")
 while 1:
 line = fd.readline()
@@ -136,6 +155,12 @@ def db_read(self, name):
 except ValueError:
 pkg.reason = -1
 raise
+elif line == "%ARCHIVED%":
+try:
+pkg.archived = int(fd.readline().strip("\n"))
+except ValueError:
+pkg.archived = -1
+raise
 elif line == "%SIZE%" or line == "%CSIZE%":
 try:
 pkg.size = int(fd.readline().strip("\n"))
@@ -162,7 +187,7 @@ def db_read(self, name):
 filename = os.path.join(path, "files")
 if not os.path.isfile(filename):
 tap.bail("invalid db entry found (files missing) for pkg " + 
pkgname)
-return None
+return
 fd = open(filename, "r")
 while 1:
 line = fd.readline()
@@ -181,7 +206,10 @@ def db_read(self, name):
 # install
 filename = os.path.join(path, "install")
 
-return pkg
+if pkg.archived:
+self.archived.setdefault(pkgname, []).append(pkg)
+else:
+self.read_pkgcache[pkgname] = pkg
 
 #
 # db_write is used to add both 'local' and 'sync' db entries
@@ -207,6 +235,7 @@ def db_write(self, pkg):
 make_section(data, "INSTALLDATE", pkg.installdate)
   

[pacman-dev] Versioned packages

[Sergey Petrenko via pacman-dev]

Here is my attempt to solve seven years old infamous problem:
https://bugs.archlinux.org/task/16702

Patch won't solve problem out of the box, a small changes in kernel PKGBUILD
will be required, but only concerning install part.

Idea behind patch is pretty simple:
1) Configure list of packages and number of old versions pacman should try 
to preserve.
2) When upgading to new version, keep old in place, if it has no file 
conflicts with new one, and mark it as `archived`, remove oldest `archived` 
version instead.

Most of time pacman treats `archived` packages as if they aren't installed.
For now it won't check package conflicts and dependencies, only file conflicts 
with newer versions. It's only an outline of full solution, proof of concept 
to illustrate the idea.

I'd like to hear opinion of community whether this problem should be solved 
at all, or is it more like a feature of ArchLinux, and if it should, whether
such approach suits ArchLinux's philosophy.

Re: [pacman-dev] [PATCH v8 01/12] bacman: allow for multiple packages as arguments

[Allan McRae]

On 05/09/16 03:37, Gordian Edenhofer wrote:
> On Sun, 2016-09-04 at 11:21 -0500, Doug Newgard wrote:
>> On Sun,  4 Sep 2016 18:13:56 +0200
>> Gordian Edenhofer  wrote:
>>
>>>
>>> To enable the creation of multiple packages with one command move
>>> the
>>> assembly process into its own function.
>>>
>>
>> Why do you keep sending these? Allan has already said they will not
>> be merged.
> 
> Allen objected parallelization as far as I understood him not the
> patches as a whole. What harm should a man page and the use of
> parseopts bring?
> Most of the recent mails were in reply to proposed amendments and the
> patchset should be properly split up into different parts by now to
> allow for cherry-picking the parts that are acceptable. Since I moved
> some things into the wrong commit when I divided 3 patches into 12 I
> had to submit it a couple of more times and I am sorry for spamming the
> mailing list with those mails.
> Furthermore until recently I was still fighting for parallelization but
> after talking to Allen on IRC I realised this is a lost case. Therefore
> upon request I will submit the patches once more without support
> for parallelization.
>

Correct - most changes will be pulled from this patchset.  Just not the
parallelization patches.  I will pull the other patches to my patchqueue
branch soon.

Allan