Re: [Pkg-postgresql-public] [pgsql-pkg-debian] Debian packaging of the q3c and pgsphere extensions

[Christoph Berg]

Re: Ole Streicher 2017-08-17 
> Markus replied to me and reminded me that I shouldn't use the tarball
> but the current git repository for the moment.
> 
> What I did, resulting in successful tests. Since Lintian didn't complain
> (and for me personally everything looks good), I uploaded the package;
> it should appear on NEW soon (which has a long queue in the moment,
> unfortunately).
> 
> Thank you very much for the help, and I am open for more comments :-)

Hi,

nice :)

I've just triggered a build on the apt.postgresql.org jenkins.
Unfortunately, pgsphere is failing the regression tests on ppc64el due
to floating point differences (e.g. 0 vs -2.38158728306e-17). This is
likely just different rounding, but this looks like a real bug:

00:01:42.739   SELECT '( 0h 2m 30s ,-90d 0m 0s)'::spoint;
00:01:42.739 ! spoint
00:01:42.739 ! --
00:01:42.739 !  (  0d 37m 30s , -90d  0m 0s)

00:01:42.743   SELECT '( 0h 2m 30s ,-90d 0m 0s)'::spoint;
00:01:42.743 ! spoint
00:01:42.743 ! --
00:01:42.743 !  (  0d 37m 30s , - 0d  0m -8262s)

The full build logs are at
https://pgdgbuild.dus.dg-i.net/job/pgsphere-binaries/1/
https://pgdgbuild.dus.dg-i.net/job/pgsphere-binaries/1/architecture=ppc64el,distribution=sid/console

(Needs a certificate from sso.debian.org for access)

Christoph

___
Pkg-postgresql-public mailing list
Pkg-postgresql-public@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-postgresql-public

Re: [Pkg-postgresql-public] [pgsql-pkg-debian] Debian packaging of the q3c and pgsphere extensions

[Ole Streicher]

Hi Christoph, Markus,
(Cc: mailing list)

On 07.08.2017 21:29, Christoph Berg wrote:
> Re: Ole Streicher 2017-08-07 
>> For pgsphere, I had less success; the tests fail currently. For me
>> (being still a strawmen), this looks like an upstream problem, so I am
>> hoping for a response from there. If you like, you can however have a
>> look as well; just to double-check that it is not a packaging problem:
> 
> I'm just browsing the source - I think core of the problem is that
> pgsphere was last updated in 2010 when PostgreSQL 9.0 was current -
> that was the last version without "CREATE EXTENSION" support

Markus replied to me and reminded me that I shouldn't use the tarball
but the current git repository for the moment.

What I did, resulting in successful tests. Since Lintian didn't complain
(and for me personally everything looks good), I uploaded the package;
it should appear on NEW soon (which has a long queue in the moment,
unfortunately).

Thank you very much for the help, and I am open for more comments :-)

Best regards

Ole

___
Pkg-postgresql-public mailing list
Pkg-postgresql-public@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-postgresql-public

Re: [Pkg-postgresql-public] [pgsql-pkg-debian] Debian packaging of the q3c and pgsphere extensions

[Christoph Berg]

Re: To Ole Streicher 2017-08-07 <20170807193234.2tubkvnei75yq...@msg.df7cb.de>
> Sorry I spoke too soon - sid/amd64 works, but ppc64el has problems:
> 
> 00:06:24.415 cat sql/ellipse.sql | psql q3c_test > results/ellipse.out
> 00:06:31.011 diff results/ellipse.out expected/ellipse.expected
> 00:06:31.012 2838c2838
> 00:06:31.012 <804
> 00:06:31.012 ---
> 00:06:31.012 >803

The problem is in pgq3c_ellipse_query_it/q3c_ellipse_query. I haven't
debugged anything, but my bet would be on floating point differences
on ppc64el, maybe because of floating point contraction (fmadd, fused
multiply add). I've had such a case before, but there only
Ubuntu/ppc64el was affected.

Christoph

___
Pkg-postgresql-public mailing list
Pkg-postgresql-public@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-postgresql-public

Re: [Pkg-postgresql-public] [pgsql-pkg-debian] Debian packaging of the q3c and pgsphere extensions

[Christoph Berg]

Re: To Ole Streicher 2017-08-07 <20170807192924.lzdjji4poyqmc...@msg.df7cb.de>
> The tests seem to work for the apt.postgresql.org jenkins as well.
> (Still building, the tests are pretty slow and there's 5 "9.x"
> versions to cover.)

Sorry I spoke too soon - sid/amd64 works, but ppc64el has problems:

00:06:24.415 cat sql/ellipse.sql | psql q3c_test > results/ellipse.out
00:06:31.011 diff results/ellipse.out expected/ellipse.expected
00:06:31.012 2838c2838
00:06:31.012 <804
00:06:31.012 ---
00:06:31.012 >803
00:06:31.013 Makefile:37: recipe for target 'test' failed

The full build log is at
https://pgdgbuild.dus.dg-i.net/job/postgresql-q3c-binaries/1/
https://pgdgbuild.dus.dg-i.net/job/postgresql-q3c-binaries/1/architecture=ppc64el,distribution=stretch/console

(Needs a SSO certificate from sso.debian.org)

Christoph

___
Pkg-postgresql-public mailing list
Pkg-postgresql-public@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-postgresql-public

Re: [Pkg-postgresql-public] [pgsql-pkg-debian] Debian packaging of the q3c and pgsphere extensions

[Christoph Berg]

Re: Ole Streicher 2017-08-07 
> > ... and a minimal debian/tests/control file. (See pg_buildext(1))
> 
> Ah, so there are no built time checks possible "just" CI test, right?

Unfortunately, right. The problem is (among others) that "CREATE
EXTENSION" wants to read the $extension.control file from
/usr/share/postgresql/$version/extension, and that part is not
configurable. (There's more path problems, but the others should be
possible to work around without patching the PostgreSQL server.)

> The tests worked for me, so I didn't change this.

Not sure what was wrong on my notebook here, but that doesn't really
matter anyway :)

The tests seem to work for the apt.postgresql.org jenkins as well.
(Still building, the tests are pretty slow and there's 5 "9.x"
versions to cover.)

> Thanks for you help! Everything seems to work now, so I uploaded q3c:
> 
> https://ftp-master.debian.org/new/postgresql-q3c_1.5.0-1.html

Nice :)

> For pgsphere, I had less success; the tests fail currently. For me
> (being still a strawmen), this looks like an upstream problem, so I am
> hoping for a response from there. If you like, you can however have a
> look as well; just to double-check that it is not a packaging problem:

I'm just browsing the source - I think core of the problem is that
pgsphere was last updated in 2010 when PostgreSQL 9.0 was current -
that was the last version without "CREATE EXTENSION" support, so it's
still using the old style of directly loading the function definitions
into the server, see sql/init.sql: "\i pg_sphere.sql". 9.0 is EOL, so
you don't need to support that style anymore.

Because of the out-of-tree build, pg_sphere.sql is not in $PWD, so
psql (or pg_regress) doesn't find it when running the tests:
regression.diffs says "pg_sphere.sql: No such file or directory".
(The PostgreSQL project still likes "context" diffs for this...)

The ideal way forward would be if you convert the module to a proper
extension. Basically you just need to write a trivial pg_sphere.control
file (see q3c for an example), and install pg_sphere.sql as
pg_sphere--1.sql. (The number there is sort of the SONAME of the
extension and might or might not correlate with the upstream version
number.)

sql/init.sql could then simply say "CREATE EXTENSION pg_sphere;".
(And expected/init.sql needs to match.)

If you expect a non-trivial number of users to have installed
pg_sphere using the "\i" method, you could also provide a
pg_sphere--unpackaged--1.sql file to allow "importing" the module
objects into an extension using "ALTER EXTENSION ADD" commands. The PG
docs cover that case as well:

https://www.postgresql.org/docs/9.6/static/extend-extensions.html

Christoph

___
Pkg-postgresql-public mailing list
Pkg-postgresql-public@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-postgresql-public

Re: [Pkg-postgresql-public] [pgsql-pkg-debian] Debian packaging of the q3c and pgsphere extensions

[Ole Streicher]

Hi Christoph,

On 06.08.2017 19:51, Christoph Berg wrote:
> Re: Ole Streicher 2017-08-04 <1c160beb-9ece-000c-628f-77a6f19db...@debian.org>
>> I have polished both packages, with the repository still in the
>> debian-science section (but I will move them to pkg-postgresql as soon
>> as I become a member). They look fine to me now (except tests; see
>> below), but could you have a look there?
> 
> Alioth says you are a pkg-postgresql member.

Yes, it works now, and both package repositories are moved (with
symlinks for some time).

> ... and a minimal debian/tests/control file. (See pg_buildext(1))

Ah, so there are no built time checks possible "just" CI test, right?

> A real problem is that the Makefile doesn't install the
> scripts/q3c--1.5.0.sql file because of its use of $(wildcard) - you'll
> need to replace that with a static list of files to install, or move
> from "pg_buildext build/install" to "pg_buildext loop".

Fixed.

> The testsuite seems to almost work except for one diff here:
> 
> diff results/version.out expected/version.expected
> 3c3
> <  f
> ---
>>  t

The tests worked for me, so I didn't change this.

> The actual debian/tests/installcheck file could be something like:
> 
> #!/bin/sh
> for v in $(pg_buildext supported-versions); do
>   LC_ALL=C pg_virtualenv -v $v make test
> done

Thanks for you help! Everything seems to work now, so I uploaded q3c:

https://ftp-master.debian.org/new/postgresql-q3c_1.5.0-1.html

For pgsphere, I had less success; the tests fail currently. For me
(being still a strawmen), this looks like an upstream problem, so I am
hoping for a response from there. If you like, you can however have a
look as well; just to double-check that it is not a packaging problem:

https://anonscm.debian.org/cgit/pkg-postgresql/pgsphere.git/

Best

Ole

___
Pkg-postgresql-public mailing list
Pkg-postgresql-public@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-postgresql-public

Re: [Pkg-postgresql-public] [pgsql-pkg-debian] Debian packaging of the q3c and pgsphere extensions

[Christoph Berg]

Re: Ole Streicher 2017-08-04 <1c160beb-9ece-000c-628f-77a6f19db...@debian.org>
> I have polished both packages, with the repository still in the
> debian-science section (but I will move them to pkg-postgresql as soon
> as I become a member). They look fine to me now (except tests; see
> below), but could you have a look there?

Alioth says you are a pkg-postgresql member.

> > I'd also like to include the packages on apt.postgresql.org.
> 
> Sure, however this is out of my own scope.

I'll take care of that part - the build system there is just
rebuilding packages either from git or from "apt-get source", so
getting the packages ready in Debian first is a natural step to do
before.

> > Fwiw, I'd strongly recommend to add a debian/tests/ testsuite (see
> > also pg_buildext(1)), even if it's just a "create extension q3c;", so
> > we have some test coverage.
> 
> I'd love to. Especially q3c has this already enabled. However I don't
> know how to enable a test postgresql database server. When I just add
> postgresql-client to d/control[.in], I get
> 
> createdb q3c_test
> createdb: could not connect to database template1: could not connect to
> server: No such file or directory
>   Is the server running locally and accepting
>   connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
> Makefile:37: recipe for target 'test' failed
> 
> during the build. Do you have any hints here?

pg_virtualenv. "pg_buildext installcheck" will use it to create
throw-away database clusters for testing.

Unless you need to do special setup steps, all you'd need is:

debian/tests/installcheck:
#!/bin/sh
pg_buildext installcheck

... and a minimal debian/tests/control file. (See pg_buildext(1))


So much for the generic hints - I've taken a look at the actual
q3c package in the meantime :)

Firstly, upstream seems to work hard on avoiding pg_regress - the
whole "diff" logic in the Makefile could likely be avoided by just
putting

REGRESS = ang2ipix area cone_join_rev cone ellipse join poly version

into the Makefile. "make installcheck" would then take care of all the
work. (But it's probably not our job as packagers to fix that...)

A real problem is that the Makefile doesn't install the
scripts/q3c--1.5.0.sql file because of its use of $(wildcard) - you'll
need to replace that with a static list of files to install, or move
from "pg_buildext build/install" to "pg_buildext loop".

The testsuite seems to almost work except for one diff here:

diff results/version.out expected/version.expected
3c3
<  f
---
>  t

The actual debian/tests/installcheck file could be something like:

#!/bin/sh
for v in $(pg_buildext supported-versions); do
  LC_ALL=C pg_virtualenv -v $v make test
done

Christoph

___
Pkg-postgresql-public mailing list
Pkg-postgresql-public@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-postgresql-public

Re: [Pkg-postgresql-public] [pgsql-pkg-debian] Debian packaging of the q3c and pgsphere extensions

[Ole Streicher]

Hi Christoph,

thank you very much for your reply with a first help.


On 03.08.2017 11:25, Christoph Berg wrote:
> Florian was talking to me about these some years back, iirc we were
> close to getting the packages polished up, but then communication got
> stuck at some point. Thanks for taking this up again!

I have polished both packages, with the repository still in the
debian-science section (but I will move them to pkg-postgresql as soon
as I become a member). They look fine to me now (except tests; see
below), but could you have a look there?

> I'd also like to include the packages on apt.postgresql.org.

Sure, however this is out of my own scope.

> pg_buildext(1) and dh_make_pgxs(1) (both from postgresql-server-dev-all)
> should provide a working skeleton at least for the case of extension-only
> packages.

Yea, pg_buildext is nice.

> Fwiw, I'd strongly recommend to add a debian/tests/ testsuite (see
> also pg_buildext(1)), even if it's just a "create extension q3c;", so
> we have some test coverage.

I'd love to. Especially q3c has this already enabled. However I don't
know how to enable a test postgresql database server. When I just add
postgresql-client to d/control[.in], I get

createdb q3c_test
createdb: could not connect to database template1: could not connect to
server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Makefile:37: recipe for target 'test' failed

during the build. Do you have any hints here?

Best regards

Ole


___
Pkg-postgresql-public mailing list
Pkg-postgresql-public@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-postgresql-public

Re: [Pkg-postgresql-public] [pgsql-pkg-debian] Debian packaging of the q3c and pgsphere extensions

[Christoph Berg]

Re: Ole Streicher 2017-07-31 
> I am posting to both lists since I am unsure where the best place is. I
> am however not subscribed to any of them, so please keep in Cc for the
> moment:

Hi Ole,

both lists are fine.

> As a dependency of the Virtual Observatory "DACHS" package [1], which I
> am planning to package, I need to have the "q3c" [2] and "pgsphere" [3]
> packages in Debian. All packages already exist in a separate repository
> [4]; the initial packaging was done by Florian Rothmaier.

Florian was talking to me about these some years back, iirc we were
close to getting the packages polished up, but then communication got
stuck at some point. Thanks for taking this up again!

I'd also like to include the packages on apt.postgresql.org.

> However, I am very new to Postgresql extension packaging, and therefore
> I would ask to get some useful hints here. Especially: Is there a Debian
> policy for postgresql extensions that documents the preferred way for
> packaging?

pg_buildext(1) and dh_make_pgxs(1) (both from postgresql-server-dev-all)
should provide a working skeleton at least for the case of extension-only
packages.

> A preferred git repository? Or is the Debian Science git
> repository OK?

Any git repo works.

> Are extensions dependent on the postgresql version, and
> if yes, how should I create the package to enable smooth transitions?

The server major version is encoded in the .so files built and the
path names, so they are dependent, yes.

> What are the preferred names for the packages?

postgresql-9.6-q3c etc.

> I would be glad to get some initial startup here (maybe a simple
> template or so)?

See pg_buildext(1) and dh_make_pgxs(1).

The existing

https://anonscm.debian.org/cgit/debian-science/packages/q3c.git/
https://anonscm.debian.org/cgit/debian-science/packages/pgsphere.git/

repositories seem mostly fine even by today's packaging standards
(maybe some of the extra clutter in debian/rules could be removed), so
you should probably go from there.

Fwiw, I'd strongly recommend to add a debian/tests/ testsuite (see
also pg_buildext(1)), even if it's just a "create extension q3c;", so
we have some test coverage.

Thanks,
Christoph


signature.asc
Description: PGP signature
___
Pkg-postgresql-public mailing list
Pkg-postgresql-public@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-postgresql-public