Re: [HACKERS] Linking libpq statically to libssl

2017-11-03 Thread Peter Eisentraut
On 11/2/17 19:52, Stephen Frost wrote:
> Uh, libpq doesn't actually have symbol versioning, at least not on the
> installed Ubuntu packages for PG10, as shown by objdump -T :

You're right, I was dreaming.  But in any case, he would need symbol
versioning of libssl.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Linking libpq statically to libssl

2017-11-03 Thread Stephen Frost
* Stephen Frost (sfr...@snowman.net) wrote:
> and we've certainly not spent effort that I've seen to try to actually
> make libpq work when multiple versions of libpq are linked into the same
> running backend.

... errr, same running application, that is, not backend.

Thanks!

Stephen


signature.asc
Description: Digital signature


Re: [HACKERS] Linking libpq statically to libssl

2017-11-02 Thread Stephen Frost
Peter,

* Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote:
> On 10/27/17 08:24, Daniele Varrazzo wrote:
> > I have a problem building binary packages for psycopg2. Binary
> > packages ship with their own copies of libpq and libssl;
> 
> Aside from the advice of "don't do that" ...
> 
> > however if
> > another python package links to libssl the library will be imported
> > twice with conflicting symbols, likely resulting in a segfault (see
> > https://github.com/psycopg/psycopg2/issues/543). This happens e.g. if
> > a python script both connects to postgres and opens an https resource.
> 
> ... the standard solutions to this problem are symbol versioning and
> linker flags to avoid making all symbols globally available.  libpq has
> symbol versioning.  Maybe the libssl you are using does not.  Also, for
> example, if you dlopen() with RTLD_LOCAL, the symbols will not be
> globally available, so there should be no conflicts.

Uh, libpq doesn't actually have symbol versioning, at least not on the
installed Ubuntu packages for PG10, as shown by objdump -T :

00011d70 gDF .text  0054  Base PQconnectStart

and we've certainly not spent effort that I've seen to try to actually
make libpq work when multiple versions of libpq are linked into the same
running backend.

I addressed that in my comments also and I don't believe you could
guarantee that things would operate correctly even with symbol
versioning being used.  Perhaps if you versioned your symbols with
something wildly different from what's on the system and hope that
what's on the system never ends up with that version then maybe it'd
work, but that's quite far from the intended use-case of symbol
versioning.

> This needs cooperation from various different parties, and the details
> will likely be platform dependent.  But it's generally a solved problem.

Right, it's solved when there's one source of truth for the library
which is being maintained consistently and carefully.  That allows
multiple versions of libraries to be installed concurrently and linked
into a running process at the same time, because the group maintaining
those different versions of the library make sure that the symbols are
versioned with appropriate versions and they make sure to not break ABI
for those symbols unless they are also changing the version and putting
out a new version.

This was all solved with GLIBC a very long time ago, but it's a heck of
a lot of work to get it right and correct, and that's when you've got a
bunch of people who work on libraries all the time and carefully control
all of the versions which are installed on the OS in coordination.
Trying to do so when you can't control what's happing with the other
library strikes me as highly likely to result in a whole lot of
difficulties.

Thanks!

Stephen


signature.asc
Description: Digital signature


Re: [HACKERS] Linking libpq statically to libssl

2017-11-02 Thread Peter Eisentraut
On 10/27/17 08:24, Daniele Varrazzo wrote:
> I have a problem building binary packages for psycopg2. Binary
> packages ship with their own copies of libpq and libssl;

Aside from the advice of "don't do that" ...

> however if
> another python package links to libssl the library will be imported
> twice with conflicting symbols, likely resulting in a segfault (see
> https://github.com/psycopg/psycopg2/issues/543). This happens e.g. if
> a python script both connects to postgres and opens an https resource.

... the standard solutions to this problem are symbol versioning and
linker flags to avoid making all symbols globally available.  libpq has
symbol versioning.  Maybe the libssl you are using does not.  Also, for
example, if you dlopen() with RTLD_LOCAL, the symbols will not be
globally available, so there should be no conflicts.

This needs cooperation from various different parties, and the details
will likely be platform dependent.  But it's generally a solved problem.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Linking libpq statically to libssl

2017-10-30 Thread Stephen Frost
Daniele,

* Daniele Varrazzo (daniele.varra...@gmail.com) wrote:
> On Fri, Oct 27, 2017 at 2:37 PM, Tom Lane  wrote:
> > Daniele Varrazzo  writes:
> >> I have a problem building binary packages for psycopg2. Binary
> >> packages ship with their own copies of libpq and libssl; however if
> >> another python package links to libssl the library will be imported
> >> twice with conflicting symbols, likely resulting in a segfault (see
> >> https://github.com/psycopg/psycopg2/issues/543). This happens e.g. if
> >> a python script both connects to postgres and opens an https resource.
> >
> > Basically, you're doing it wrong.  Shipping your own copy of libssl,
> > rather than depending on whatever packaging the platform provides,
> > is just asking for pain --- and not only of this sort.  You're also
> > now on the hook to update your package whenever libssl fixes a bug
> > or a security vulnerability, which happens depressingly often.
> >
> > The same applies to libpq, really.  You don't want to be in the
> > business of shipping bits that you are not the originator of.
> >
> > When I worked at Red Hat, there was an ironclad policy against
> > building packages that incorporated other packages statically.
> > I would imagine that other distros have similar policies for
> > similar reasons.  Just because you *can* ignore those policies
> > doesn't mean you *should*.
> 
> Distros do compile the library from source and against the system
> package, and everyone using the package directly can still do so; the
> binary packages are only installed by the Python package manager.

Which, frankly, is why everyone having their own package manager that
ignores the OS package manager is actually rather horrible.  Obviously,
it's done extensively, but it's outright horrible.

> Psycopg is more complex to install than the average Python package (it
> needs python and postgres dev files, pg_config available somewhere
> etc) and a no-dependencies package provides a much smoother
> experience. It also happens that the libpq and libssl versions used
> tend to be more up-to-date than the system one (people can already use
> the new libpq 10 features without waiting for debian packages).

Having randomly different versions of libraries installed through two
different package managers on the same system is not an improvement for
users.  Further, it's not like there aren't already properly built PG10
packages- they were there from the day PG10 was released, and users can
install them using the OS package manager and against the OS provided
version of all the various libraries.

> I am confronted with the reality of Python developers as of 201x's,
> and shipping binary packages has proven generally a positive feature,
> even factoring in the need of shipping updated binary packages when
> the need arises. The benefit of a simple-to-use library is for the
> Postgres project at large, it is not for my personal gain. So while I
> know the shortcomings of binary packages and static libraries, I would
> still be interested in knowing the best way to fix the problem in the
> subject. Feel free to write in private if you want to avoid the public
> shaming.

The way to fix the problem is to not use two different versions of the
same library in the same binary, which you aren't going to be able to
accomplish when you're forcibly pulling in a different version through
your own binary package.  At best you could try to use symbol versioning
to try and differentiate your symbols from those of the system one, but
that would depend on if the system level library is doing symbol
versioning or not and then you'd still have to figure out what to do
when the OS level package updates and possibly ends up with symbols
*and* versions conflicting.  Of course, these issues are generally (or
should be) handled already by the OS level library package managers and
it's no simple thing to do, but it's how things like multiple libdb
versions can be available and even linked into the same running binaries
and things still work there.  Basically, if you can't control both
versions you're just setting yourself (and our users) up for failure, if
not now then in the future.

In short, figure out how to have a completely different OS that's only
provided through your package manager, or get along with the packages
and versions as provided through the OS system (or provide your own
updated versions of the OS packages and get them installed that matches
what your packages are built against).

Thanks!

Stephen


signature.asc
Description: Digital signature


Re: [HACKERS] Linking libpq statically to libssl

2017-10-27 Thread Daniele Varrazzo
On Fri, Oct 27, 2017 at 2:37 PM, Tom Lane  wrote:
> Daniele Varrazzo  writes:
>> I have a problem building binary packages for psycopg2. Binary
>> packages ship with their own copies of libpq and libssl; however if
>> another python package links to libssl the library will be imported
>> twice with conflicting symbols, likely resulting in a segfault (see
>> https://github.com/psycopg/psycopg2/issues/543). This happens e.g. if
>> a python script both connects to postgres and opens an https resource.
>
> Basically, you're doing it wrong.  Shipping your own copy of libssl,
> rather than depending on whatever packaging the platform provides,
> is just asking for pain --- and not only of this sort.  You're also
> now on the hook to update your package whenever libssl fixes a bug
> or a security vulnerability, which happens depressingly often.
>
> The same applies to libpq, really.  You don't want to be in the
> business of shipping bits that you are not the originator of.
>
> When I worked at Red Hat, there was an ironclad policy against
> building packages that incorporated other packages statically.
> I would imagine that other distros have similar policies for
> similar reasons.  Just because you *can* ignore those policies
> doesn't mean you *should*.

Distros do compile the library from source and against the system
package, and everyone using the package directly can still do so; the
binary packages are only installed by the Python package manager.
Psycopg is more complex to install than the average Python package (it
needs python and postgres dev files, pg_config available somewhere
etc) and a no-dependencies package provides a much smoother
experience. It also happens that the libpq and libssl versions used
tend to be more up-to-date than the system one (people can already use
the new libpq 10 features without waiting for debian packages).

I am confronted with the reality of Python developers as of 201x's,
and shipping binary packages has proven generally a positive feature,
even factoring in the need of shipping updated binary packages when
the need arises. The benefit of a simple-to-use library is for the
Postgres project at large, it is not for my personal gain. So while I
know the shortcomings of binary packages and static libraries, I would
still be interested in knowing the best way to fix the problem in the
subject. Feel free to write in private if you want to avoid the public
shaming.

-- Daniele


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Linking libpq statically to libssl

2017-10-27 Thread Tom Lane
Daniele Varrazzo  writes:
> I have a problem building binary packages for psycopg2. Binary
> packages ship with their own copies of libpq and libssl; however if
> another python package links to libssl the library will be imported
> twice with conflicting symbols, likely resulting in a segfault (see
> https://github.com/psycopg/psycopg2/issues/543). This happens e.g. if
> a python script both connects to postgres and opens an https resource.

Basically, you're doing it wrong.  Shipping your own copy of libssl,
rather than depending on whatever packaging the platform provides,
is just asking for pain --- and not only of this sort.  You're also
now on the hook to update your package whenever libssl fixes a bug
or a security vulnerability, which happens depressingly often.

The same applies to libpq, really.  You don't want to be in the
business of shipping bits that you are not the originator of.

When I worked at Red Hat, there was an ironclad policy against
building packages that incorporated other packages statically.
I would imagine that other distros have similar policies for
similar reasons.  Just because you *can* ignore those policies
doesn't mean you *should*.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Linking libpq statically to libssl

2017-10-27 Thread Daniele Varrazzo
Hello,

I have a problem building binary packages for psycopg2. Binary
packages ship with their own copies of libpq and libssl; however if
another python package links to libssl the library will be imported
twice with conflicting symbols, likely resulting in a segfault (see
https://github.com/psycopg/psycopg2/issues/543). This happens e.g. if
a python script both connects to postgres and opens an https resource.

A solution to work around the problem could be to change libssl
symbols names, in the binaries or in the source code
(https://github.com/psycopg/psycopg2-wheels/issues/7). But maybe a
simpler workaround could be just to build libpq linking to libssl
statically. Is this possible?

...and other libs too I guess, because the problem was found on libssl
but conflicts with other libs is possible too, ldap comes to mind...

Any help to solve the problem would be appreciated. Thank you very much.

-- Daniele


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers