[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Nobuto Murata
A deployment method improvement in the field will be tracked as a private bug LP: #1889498. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1939898 Title: Unnatended postgresql-12 upgrade caused MAAS

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Nobuto Murata
Closing MAAS task since MAAS just connects to PostgreSQL through tcp connections. One correction to my previous statement: $ sudo lsof / | grep plpgsql.so postgres 21822 postgres memREG 252,1 202824 1295136 /usr/lib/postgresql/12/lib/plpgsql.so postgres 21948 postgres

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Christian Ehrhardt 
Thanks Christoph for chiming in and a second voice that confirms what I said. So to summarize: - we have found the root cause to be sure what is going on (stored procedures triggered late lib loading of new on-disk .so files by an old instance of the server) - Packaging takes care of

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Christoph Berg
Fwiw the use-case of not restarting after installing a minor upgrade is not supported by PostgreSQL upstream. The proper fix is to restart, or to hold off the upgrade until the time when restarting is possible. -- You received this bug notification because you are a member of Ubuntu Bugs, which

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Christian Ehrhardt 
Testcase derived: $ sudo -u postgres psql -c 'CREATE DATABASE test;' $ sudo -u postgres psql -d test -c 'CREATE PROCEDURE test() LANGUAGE plpgsql AS $$ BEGIN ROLLBACK; END; $$;'; $ sudo -u postgres psql -c 'DROP DATABASE test;' Good: CREATE DATABASE CREATE PROCEDURE DROP DATABASE Bad: CREATE

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Christian Ehrhardt 
adam got in touch (thnaks) and we got sudo snap run --shell maas -c 'maas-region shell' That drops me in "another" python shell but I couldn't get anything more useful from it. Their theory actually matches mine closely which is around stored procedures which maas seems to use heavily. That

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Christian Ehrhardt 
mdeslaur suggested that it uses pyscopg2 And indeed I can find that in the snap: $ find /snap/maas/ -type d| grep psycopg2 /snap/maas/15003/usr/lib/python3/dist-packages/django/db/backends/postgresql_psycopg2

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Christian Ehrhardt 
Ok, any SQL issued from maas seems to fail: $ maas createadmin --username test1 --password test2 --email t...@test.com leads to root@f-pgdebug:/# 2021-08-16 14:35:39.130 GMT [13818] ERROR: could not load library "/usr/lib/postgresql/12/lib/plpgsql.so": /usr/lib/postgresql/12/lib/plpgsql.so:

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Christian Ehrhardt 
Ok, I've set up a system as instructed - thanks again Nobuto! Using maas from the snap 3.0/stable: 3.0.0-10029-g.986ea3e452021-06-22 (15003) 141MB - And postgresql (old) from packaging. root@f-pgdebug:/# dpkg -l | grep postgres ii postgresql 12+214

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread mastier1
Damn, you were right Nobuto. I completely forgot that postgresql is managed by Corosync $ crm restource restart ms_pgsql did the trick Anyway that is something need to be considered. And propably other deployments will fail in the process -- You received this bug notification because you are

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Christian Ehrhardt 
Thank Nobuto, I agree if not restarted there could be issues. In addition to the general habit of restarting services if possible there now also is the needrestart service in newer releases to further nudge users/admins to realize that. For databases all of that isn't so easy - as Nobuto

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Björn Tillenius
It feels to me, that this bug isn't related to MAAS itself, or the fact that it's a snap. MAAS is a confined snap, so upgrading the postgres package on the host system shouldn't have any effect at all. If it does indeed has an affect, we'd need to involve the snapd team. But I don't think that's

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Nobuto Murata
The only scenario I can think of is NOT restarting postgres after the package update. This could happen when postgres process is managed outside of init(systemd) such as pacemaker, etc. for HA purposes. $ sudo lsof / | grep plpgsql.so postgres 21822 postgres DELREG 252,1

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Adam Collard
Have tried to reproduce this with a minimal setup, but haven't managed to just yet. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1939898 Title: Unnatended postgresql-12 upgrade caused MAAS

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Adam Collard
I hit this in my development environment with no MAAS snap involved: focal LXD container, long lived which then got updates to the various postgresql packages via unattended-upgrades - see https://paste.ubuntu.com/p/VrY4Gv8SyC/ for /var/log/apt/history.log snippet, the last two entries being me

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-16 Thread Christian Ehrhardt 
I agree to the check of mastier1: Old version $ apt download postgresql-12=12.2-4 $ dpkg -x postgresql-12_12.2-4_amd64.deb old $ nm -D old/usr/lib/postgresql/12/lib/plpgsql.so | grep EnsurePortalSnapshotExists New version $ apt download postgresql-12=12.8-0ubuntu0.20.04.1 $ dpkg -x

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-13 Thread Marc Deslauriers
I suspect the MAAS snap has embedded some postgresql components while relying on some other components from the system, and this mismatch is causing the issue loading the newer shared library... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1939898] Re: Unnatended postgresql-12 upgrade caused MAAS internal error

2021-08-13 Thread mastier1
The symbol appears as dynamic symbol in 12.8, was not there in 12.7 12.7 https://pastebin.canonical.com/p/SXkBCDWvsj/ 12.8 $ nm -D /usr/lib/postgresql/12/lib/plpgsql.so|grep Exis U EnsurePortalSnapshotExists -- You received this bug notification because you are a member of