Bug#857770: [Pkg-postgresql-public] Bug#857770: src:postgresql-debversion: please provide a unversioned binary package

[Christoph Berg]

Re: Ole Streicher 2017-09-25 <765c2141-8d1b-44c6-845b-6375d8499...@debian.org>
> This can be addressed by introducing Postgresql-versioned Provides. If
> the q3c package would support f.e. 9.6 and 10, it can have
> 
> Provides: postgresql-9.6-q3c, postgresql-10-q3c
> 
> If you then need q3c for a specific version, you can install/depend on
> this specific version. This also allows a smooth transition between the
> multi-binary-package approach and the single binary package.

Aye, didn't think of that aspect yet.

> > Maybe we could just claim that the user should upgrade their database
> > at the same time, and for many extensions this will not be a problem.
> > But for extensions providing datatypes (I think that includes q3c),
> > the .so module is needed for dumping the old data, so at least
> > dump-restore upgrades are affected. pg_upgrade should still work,
> > though. (Making pg_upgradecluster use that by default is another
> > story.)
> 
> This may indeed need some work; however I need some advice here.

The problem is that at no point in time both versions are installed,
so upgrading a PostgreSQL cluster via online dump-restore won't work.
We could make that work by supporting both the old and the new
PostgreSQL version in the next release, but that would double the
maintenance effort on the PostgreSQL server packages side. PostgreSQL
major versions are supported upstream for 5 years, but we are
regularly running out of support even with one version in
oldoldstable-lts, that would be much worse if there were two versions
(likely released 2 years apart).

Christoph

Bug#857770: [Pkg-postgresql-public] Bug#857770: src:postgresql-debversion: please provide a unversioned binary package

[Ole Streicher]

Hi Christoph,

On 23.09.2017 21:41, Christoph Berg wrote:
> Re: Ole Streicher 2017-09-22 
>> I would just do this for my own packages (pqsphere and q3c), which would
>> allow to collect experiences; but ofcourse I don't want to shoot into
>> your feet with the packaging at the postgresql site.
> 
> The plan seems sensible, so please go ahead, it will be an interesting
> experiment. The plethora of mini packages we have now isn't ideal.

I just uploaded q3c as a single binary package. pgsphere will follow
after some discussion with the main users (hi Markus :-) )

> My main concern is supporting upgrades cleanly. Imagine "your" package
> had already been around in jessie, and the user had postgresql-9.4 +
> postgresql-q3c installed where q3c provided the 9.4 extension. Now the
> user upgrades to stretch, gets postgresql-9.6 installed (e.g. via
> postgresql.deb), the 9.4 cluster will still work, but the upgraded
> postgresql-q3c package from stretch suddenly doesn't provide the 9.4
> anymore, but only 9.6, rendering q3c broken in the 9.4 cluster.

This can be addressed by introducing Postgresql-versioned Provides. If
the q3c package would support f.e. 9.6 and 10, it can have

Provides: postgresql-9.6-q3c, postgresql-10-q3c

If you then need q3c for a specific version, you can install/depend on
this specific version. This also allows a smooth transition between the
multi-binary-package approach and the single binary package.

This is already implemented in the uploaded version.

> Maybe we could just claim that the user should upgrade their database
> at the same time, and for many extensions this will not be a problem.
> But for extensions providing datatypes (I think that includes q3c),
> the .so module is needed for dumping the old data, so at least
> dump-restore upgrades are affected. pg_upgrade should still work,
> though. (Making pg_upgradecluster use that by default is another
> story.)

This may indeed need some work; however I need some advice here.

Best regards

Ole

Bug#857770: [Pkg-postgresql-public] Bug#857770: src:postgresql-debversion: please provide a unversioned binary package

[Christoph Berg]

Re: Ole Streicher 2017-09-22 
> Hi Christoph,
> 
> may I ping you here? Postgresql-10 just arrived in unstable (great,
> thanks); but I would now rather like to convert to single (and
> bin-NMU-capable) packages instead of adding postgresql-10-q3c now, as it
> is done f.e. for Python packages.
> 
> I would just do this for my own packages (pqsphere and q3c), which would
> allow to collect experiences; but ofcourse I don't want to shoot into
> your feet with the packaging at the postgresql site.

The plan seems sensible, so please go ahead, it will be an interesting
experiment. The plethora of mini packages we have now isn't ideal.

(While uploading the first batch of PG10 packages, I rediscovered the
fact that the pgagent package is already using such a scheme:

Package: pgagent
XB-Postgresql-Version: ${Postgresql:Version}
Postgresql-Version: 9.2 9.3 9.4 9.5 9.6

The package is pretty small, though, pgagent's extension part is SQL
only, and version-agnostic, only the pgagent client binary needs to be
compiled, and doesn't vary between versions.)

My main concern is supporting upgrades cleanly. Imagine "your" package
had already been around in jessie, and the user had postgresql-9.4 +
postgresql-q3c installed where q3c provided the 9.4 extension. Now the
user upgrades to stretch, gets postgresql-9.6 installed (e.g. via
postgresql.deb), the 9.4 cluster will still work, but the upgraded
postgresql-q3c package from stretch suddenly doesn't provide the 9.4
anymore, but only 9.6, rendering q3c broken in the 9.4 cluster.

Maybe we could just claim that the user should upgrade their database
at the same time, and for many extensions this will not be a problem.
But for extensions providing datatypes (I think that includes q3c),
the .so module is needed for dumping the old data, so at least
dump-restore upgrades are affected. pg_upgrade should still work,
though. (Making pg_upgradecluster use that by default is another
story.)

Christoph

Bug#857770: [Pkg-postgresql-public] Bug#857770: src:postgresql-debversion: please provide a unversioned binary package

[Ole Streicher]

Hi Christoph,

may I ping you here? Postgresql-10 just arrived in unstable (great,
thanks); but I would now rather like to convert to single (and
bin-NMU-capable) packages instead of adding postgresql-10-q3c now, as it
is done f.e. for Python packages.

I would just do this for my own packages (pqsphere and q3c), which would
allow to collect experiences; but ofcourse I don't want to shoot into
your feet with the packaging at the postgresql site.

Best regards

Ole

On 30.08.2017 16:26, Ole Streicher wrote:
> Hi Christoph,
> 
> On 30.08.2017 15:50, Christoph Berg wrote:
>> Re: Ole Streicher 2017-08-30 
>> 
>>> The idea here is to have just one binary package, containing the shared
>>> libraries for all supported versions. Extensions are usually small, so
>>> combining them in one package will not hurt. So, there would no
>>> "postgresql-9.6-q3c" package anymore, d/control.in is removed, and
>>> d/control is fixed (for one release). [...]
>>
>> That is true, but it's totally different from what we have been doing
>> so far. We would need to update all packages, and providing the
>> necessary (?) transitional packages for existing users will be
>> difficult.
> 
> There is no need to update quickly: if just the dependency variable
> creation is integrated in pg_buildext, then the maintainer can still
> choose between the current approach and the single-package approach. It
> is just a matter of policy then.
> 
> Transitioning should just be possible with an additional field
> 
> Provides: postgresql-9.6-debconf, postgresql-10.0-debconf [...]
> 
> which also could be supported by a variable {postgresql:Provides},
> created similarly to {postgresql:Depends} field.
> 
>> If a PG version goes out of support, the new package version wouldn't
>> contain the module anymore, even if users are still using that PG
>> version on their system. (Think Debian dist-upgrades.)
> 
> And if a postgresql version goes out of support, then I would suppose
> that the postgresql-XX server package is removed as well? Then, they
> couldn't upgrade the package because of the missing postgresql
> dependency, and they couldn't upgrade in the old scheme.
> 
> But I must say I have no idea how upgrades for Postgresql work, there
> may be some problems.
> 
>> It would also prevent (easily) building packages against beta/devel PG
>> versions (10 and 11 at the moment). Or these packages would need to
>> include the "normal" PG versions from the normal packages, plus the
>> extra versions.
> 
> The only difference is that all builds are combined into a single
> package. So, whatever can be built in the current scheme, can also be
> built then. If "pg_buildext supported-versions" offers experimental
> packages, they are also included.
> 
> Best regards
> 
> Ole
> 

Bug#857770: [Pkg-postgresql-public] Bug#857770: src:postgresql-debversion: please provide a unversioned binary package

[Ole Streicher]

Hi Christoph,

On 30.08.2017 15:50, Christoph Berg wrote:
> Re: Ole Streicher 2017-08-30 
>> The idea here is to have just one binary package, containing the shared
>> libraries for all supported versions. Extensions are usually small, so
>> combining them in one package will not hurt. So, there would no
>> "postgresql-9.6-q3c" package anymore, d/control.in is removed, and
>> d/control is fixed (for one release). [...]
> 
> That is true, but it's totally different from what we have been doing
> so far. We would need to update all packages, and providing the
> necessary (?) transitional packages for existing users will be
> difficult.

There is no need to update quickly: if just the dependency variable
creation is integrated in pg_buildext, then the maintainer can still
choose between the current approach and the single-package approach. It
is just a matter of policy then.

Transitioning should just be possible with an additional field

Provides: postgresql-9.6-debconf, postgresql-10.0-debconf [...]

which also could be supported by a variable {postgresql:Provides},
created similarly to {postgresql:Depends} field.

> If a PG version goes out of support, the new package version wouldn't
> contain the module anymore, even if users are still using that PG
> version on their system. (Think Debian dist-upgrades.)

And if a postgresql version goes out of support, then I would suppose
that the postgresql-XX server package is removed as well? Then, they
couldn't upgrade the package because of the missing postgresql
dependency, and they couldn't upgrade in the old scheme.

But I must say I have no idea how upgrades for Postgresql work, there
may be some problems.

> It would also prevent (easily) building packages against beta/devel PG
> versions (10 and 11 at the moment). Or these packages would need to
> include the "normal" PG versions from the normal packages, plus the
> extra versions.

The only difference is that all builds are combined into a single
package. So, whatever can be built in the current scheme, can also be
built then. If "pg_buildext supported-versions" offers experimental
packages, they are also included.

Best regards

Ole

Bug#857770: [Pkg-postgresql-public] Bug#857770: src:postgresql-debversion: please provide a unversioned binary package

[Christoph Berg]

Re: Ole Streicher 2017-08-30 
> The idea here is to have just one binary package, containing the shared
> libraries for all supported versions. Extensions are usually small, so
> combining them in one package will not hurt. So, there would no
> "postgresql-9.6-q3c" package anymore, d/control.in is removed, and
> d/control is fixed (for one release). For convenience, I just attach the
> complete d/control for postgresql-q3c.
> 
> In that case, changing the supported postgresql versions will not change
> the list of binary packages, but just the dependencies, which is IMO not
> forbidden.

That is true, but it's totally different from what we have been doing
so far. We would need to update all packages, and providing the
necessary (?) transitional packages for existing users will be
difficult.

If a PG version goes out of support, the new package version wouldn't
contain the module anymore, even if users are still using that PG
version on their system. (Think Debian dist-upgrades.)

It would also prevent (easily) building packages against beta/devel PG
versions (10 and 11 at the moment). Or these packages would need to
include the "normal" PG versions from the normal packages, plus the
extra versions.

The idea is intriguing, though. Maybe these problems have cute
solutions and could be fixed.

Christoph


signature.asc
Description: PGP signature

Bug#857770: [Pkg-postgresql-public] Bug#857770: src:postgresql-debversion: please provide a unversioned binary package

[Ole Streicher]

Hi Chriostoph,

On 30.08.2017 14:03, Christoph Berg wrote:
> Re: Ole Streicher 2017-08-30 
>> If a new Postgresql version is uploaded (or an old one is removed), a
>> binNMU can be requested, resulting in a new package with the new list of
>> Postgresql objects built in. As it is done for Python or others.
> 
> BinNMUs would work for updating the ${postgresql:Depends} part. What
> doesn't work is changing the set of binaries built from
> debian/control, because it's forbidden by Debian policy. This is why
> "pg_buildext checkcontrol" raises an error if the file changes.
> Possibly we could make it smarter, but in most cases, changing the
> default version will mean the list of supported versions changes as
> well.

The idea here is to have just one binary package, containing the shared
libraries for all supported versions. Extensions are usually small, so
combining them in one package will not hurt. So, there would no
"postgresql-9.6-q3c" package anymore, d/control.in is removed, and
d/control is fixed (for one release). For convenience, I just attach the
complete d/control for postgresql-q3c.

In that case, changing the supported postgresql versions will not change
the list of binary packages, but just the dependencies, which is IMO not
forbidden.

Best regards

Ole


Source: postgresql-q3c
Section: database
Priority: optional
Maintainer: Debian PostgreSQL Maintainers 

Uploaders: Ole Streicher ,
   Markus Nullmeier 
Build-Depends: debhelper (>= 9),
   postgresql-server-dev-all
Standards-Version: 4.1.0
Homepage: http://www.sai.msu.su/~megera/wiki/SkyPixelization
Vcs-Browser: https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-q3c.git
Vcs-Git: https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-q3c.git

Package: postgresql-q3c
Architecture: any
Depends: ${postgresql:Depends},
 ${misc:Depends},
 ${shlibs:Depends}
Description: PostgreSQL extension used for indexing the sky
 Q3C, an extension for PostgreSQL, is designed for the work with large
 astronomical catalogues or any catalogs of objects on the sphere.
 .
 This extension allows a user to perform fast circular, elliptical or
 polygonal searches on the sky as well as fast cross-matches.
 .
 This package provides a module for PostgreSQL.

Bug#857770: [Pkg-postgresql-public] Bug#857770: src:postgresql-debversion: please provide a unversioned binary package

[Christoph Berg]

Re: Ole Streicher 2017-08-30 
> For Debian (single Postgresql version) this works well; I don't know
> if "pg_buildext supported-versions" returns them line by line (what I assumed)
> or space-separated (then it needs some adjustments). One should also discuss
> which Postgresql version should be the first (which installed by default).

It's line-separated.

Re the default version, the convention is that
/usr/share/postgresql-common/supported-versions returns the default
version /last/. (Try calling
  PG_SUPPORTED_VERSIONS=pgdg /usr/share/postgresql-common/supported-versions
).

I'm not sure if "pg_buildext supported-versions" preserves that
ordering in all cases; but we can fix it to do so if it's not already
the case. Then there is the question what to do if the default version
is not in the intersection of "debian/pgversions"
`/usr/share/postgresql-common/supported-versions`, which is what
`pg_buildext supported-versions` really computes.

> Ideally the dependency generation could be integrated into pg_buildext.

Right.

> If a new Postgresql version is uploaded (or an old one is removed), a
> binNMU can be requested, resulting in a new package with the new list of
> Postgresql objects built in. As it is done for Python or others.

BinNMUs would work for updating the ${postgresql:Depends} part. What
doesn't work is changing the set of binaries built from
debian/control, because it's forbidden by Debian policy. This is why
"pg_buildext checkcontrol" raises an error if the file changes.
Possibly we could make it smarter, but in most cases, changing the
default version will mean the list of supported versions changes as
well.

Christoph