Re: [Openstack-operators] [openstack-dev] RFC: Next minimum libvirt / QEMU versions for "Stein" release

[Matt Riedemann]

On 4/5/2018 3:32 PM, Thomas Goirand wrote:

If you don't absolutely need new features from libvirt 3.2.0 and 3.0.0
is fine, please choose 3.0.0 as minimum.

If you don't absolutely need new features from qemu 2.9.0 and 2.8.0 is
fine, please choose 2.8.0 as minimum.

If you don't absolutely need new features from libguestfs 1.36 and 1.34
is fine, please choose 1.34 as minimum.


New features in the libvirt driver which depend on minimum versions of 
libvirt/qemu/libguestfs (or arch for that matter) are always 
conditional, so I think it's reasonable to go with the lower bound for 
Debian. We can still support the features for the newer versions if 
you're running a system with those versions, but not penalize people 
with slightly older versions if not.


--

Thanks,

Matt

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] RFC: Next minimum libvirt / QEMU versions for "Stein" release

[Thomas Goirand]

On 04/04/2018 10:45 AM, Kashyap Chamarthy wrote:
> Answering my own questions about Debian --
> 
> From looking at the Debian Archive[1][2], these are the versions for
> 'Stretch' (the current stable release) and in the upcoming 'Buster'
> release:
> 
> libvirt | 3.0.0-4+deb9u2 | stretch
> libvirt | 4.1.0-2| buster
> 
> qemu| 1:2.8+dfsg-6+deb9u3| stretch
> qemu| 1:2.11+dfsg-1  | buster
> 
> I also talked on #debian-backports IRC channel on OFTC network, where I
> asked: 
> 
> "What I'm essentially looking for is: "How can 'stretch' users get
> libvirt 3.2.0 and QEMU 2.9.0, even if via a different repository.
> As they are proposed to be least common denominator versions across
> distributions."
> 
> And two people said: Then the versions from 'Buster' could be backported
> to 'stretch-backports'.  The process for that is to: "ask the maintainer
> of those package and Cc to the backports mailing list."
> 
> Any takers?
> 
> [0] https://packages.debian.org/stretch-backports/
> [1] https://qa.debian.org/madison.php?package=libvirt
> [2] https://qa.debian.org/madison.php?package=qemu

Hi Kashyap,

Thanks for your considering of Debian, asking me and giving enough time
for answering! Here's my thoughts.

I updated the wiki page as you suggested [1]. As i wrote on IRC, we
don't need to care about Jessie, so I removed Jessie, and added Buster/SID.

tl;dr: just skip this section & go to conclusion

backport of libvirt/QEMU/libguestfs more in details
---

I already attempted the backports from Debian Buster to Stretch.

All of the 3 components (libvirt, qemu & libguestfs) could be built
without extra dependency, which is a very good thing.

- libvirt 4.1.0 compiled without issue, though the dh_install phase
failed with this error:

dh_install: Cannot find (any matches for) "/usr/lib/*/wireshark/" (tried
in "." and "debian/tmp")
dh_install: libvirt-wireshark missing files: /usr/lib/*/wireshark/
dh_install: missing files, aborting

Without more investigation but this build log, it's likely a minor fix
in debian/*.install files to make it possible to backport the package.

- qemu 2.11 built perfectly with zero change.

- libguestfs 1.36.13 only needed to have fdisk replaced by util-linux as
build-depends (fdisk is now a separate package in Buster).

So it looks like easy to backport these 3 *AT THIS TIME*. [2]

However, without a cristal ball, nobody can tell how hard it will be to
backport these *IN A YEAR FROM NOW*.

Conclusion:
---

If you don't absolutely need new features from libvirt 3.2.0 and 3.0.0
is fine, please choose 3.0.0 as minimum.

If you don't absolutely need new features from qemu 2.9.0 and 2.8.0 is
fine, please choose 2.8.0 as minimum.

If you don't absolutely need new features from libguestfs 1.36 and 1.34
is fine, please choose 1.34 as minimum.

If you do need these new features, I'll do my best adapt. :)

About Buster freeze & OpenStack Stein backports to Debian Stretch
-

Now, about Buster. As you know, Debian doesn't have planned release
dates. Though here's the stats showing that roughly, there's a new
Debian every 2 years, and the freeze takes about 6 months.

https://wiki.debian.org/DebianReleases#Release_statistics

With this logic and considering Stretch was released last year in June,
after Stein is released, Buster will probably start its freeze. If the
Debian freeze happens later, good for me, I'll have more time to make
Stein better. But then Debian users will probably expect an OpenStack
Stein backport to Debian Stretch, and that's where it can become tricky
to backport these 3 packages.

The end
---

I hope the above isn't too long, and helps to take the best decision,
Cheers,

Thomas Goirand (zigo)

[1]
https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix#Distro_minimum_versions

[2] I'm not shouting, just highlighting the important part! :)

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] Asking for ask.openstack.org

[Jimmy McArthur]

Ian, thanks for digging in and helping sort out some of these issues!


Ian Wienand 
April 4, 2018 at 11:04 PM

We've long had problems with this host and I've looked at it before
[1]. It often drops out.

It seems there's enough interest we should dive a bit deeper. Here's
what I've found out:

askbot
--

Of the askbot site, it seems under control, except for an unbounded
session log file. Proposed [2]

root@ask:/srv# du -hs *
2.0G askbot-site
579M dist

overall
---

The major consumer is /var; where we've got

3.9G log
5.9G backups
9.4G lib

backups
---

The backup seem under control at least; we're rotating them out and we
keep 10, and the size is pretty consistently 500mb:

root@ask:/var/backups/pgsql_backups# ls -lh
total 5.9G
-rw-r--r-- 1 root root 599M Apr 5 00:03 askbotdb.sql.gz
-rw-r--r-- 1 root root 598M Apr 4 00:03 askbotdb.sql.gz.1
...

We could reduce the backup rotations to just one if we like -- the
server is backed up nightly via bup, so at any point we can get
previous dumps from there. bup should de-duplicate everything, but
still, it's probably not necessary.

The db directory was sitting at ~9gb

root@ask:/var/lib/postgresql# du -hs
8.9G .

AFAICT, it seems like the autovacuum is running OK on the busy tables

askbotdb=# select relname,last_vacuum, last_autovacuum, last_analyze, 
last_autoanalyze from pg_stat_user_tables where last_autovacuum is not 
NULL;

relname | last_vacuum | last_autovacuum | last_analyze | last_autoanalyze
--+-+---+---+---
django_session | | 2018-04-02 17:29:48.329915+00 | 2018-04-05 
02:18:39.300126+00 | 2018-04-05 00:11:23.456602+00
askbot_badgedata | | 2018-04-04 07:19:21.357461+00 | | 2018-04-04 
07:18:16.201376+00
askbot_thread | | 2018-04-04 16:24:45.124492+00 | | 2018-04-04 
20:32:25.845164+00
auth_message | | 2018-04-04 12:29:24.273651+00 | 2018-04-05 
02:18:07.633781+00 | 2018-04-04 21:26:38.178586+00
djkombu_message | | 2018-04-05 02:11:50.186631+00 | | 2018-04-05 
02:14:45.22926+00


Out of interest I did run a manual

su - postgres -c "vacuumdb --all --full --analyze"

We dropped something

root@ask:/var/lib/postgresql# du -hs
8.9G .
(after)
5.8G

I installed pg_activity and watched for a while; nothing seemed to be
really stressing it.

Ergo, I'm not sure if there's much to do in the db layers.

logs


This leaves the logs

1.1G jetty
2.9G apache2

The jetty logs are cleaned regularly. I think they could be made more
quiet, but they seem to be bounded.

Apache logs are rotated but never cleaned up. Surely logs from 2015
aren't useful. Proposed [3]

Random offline
--

[3] is an example of a user reporting the site was offline. Looking
at the logs, it seems that puppet found httpd not running at 07:14 and
restarted it:

Apr 4 07:14:40 ask puppet-user[20737]: 
(Scope(Class[Postgresql::Server])) Passing "version" to 
postgresql::server is deprecated; please use postgresql::globals instead.
Apr 4 07:14:42 ask puppet-user[20737]: Compiled catalog for 
ask.openstack.org in environment production in 4.59 seconds

Apr 4 07:14:44 ask crontab[20987]: (root) LIST (root)
Apr 4 07:14:49 ask puppet-user[20737]: 
(/Stage[main]/Httpd/Service[httpd]/ensure) ensure changed 'stopped' to 
'running'
Apr 4 07:14:54 ask puppet-user[20737]: Finished catalog run in 10.43 
seconds


Which first explains why when I looked, it seemed OK. Checking the
apache logs we have:

[Wed Apr 04 07:01:08.144746 2018] [:error] [pid 12491:tid 
140439253419776] [remote 176.233.126.142:43414] mod_wsgi (pid=12491): 
Exception occurred processing WSGI script 
'/srv/askbot-site/config/django.wsgi'.
[Wed Apr 04 07:01:08.144870 2018] [:error] [pid 12491:tid 
140439253419776] [remote 176.233.126.142:43414] IOError: failed to 
write data

... more until ...
[Wed Apr 04 07:15:58.270180 2018] [:error] [pid 17060:tid 
140439253419776] [remote 176.233.126.142:43414] mod_wsgi (pid=17060): 
Exception occurred processing WSGI script 
'/srv/askbot-site/config/django.wsgi'.
[Wed Apr 04 07:15:58.270303 2018] [:error] [pid 17060:tid 
140439253419776] [remote 176.233.126.142:43414] IOError: failed to 
write data


and the restart logged

[Wed Apr 04 07:14:48.912626 2018] [core:warn] [pid 21247:tid 
140439370192768] AH00098: pid file /var/run/apache2/apache2.pid 
overwritten -- Unclean shutdown of previous Apache run?
[Wed Apr 04 07:14:48.913548 2018] [mpm_event:notice] [pid 21247:tid 
140439370192768] AH00489: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f 
mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
[Wed Apr 04 07:14:48.913583 2018] [core:notice] [pid 21247:tid 
140439370192768] AH00094: Command line: '/usr/sbin/apache2'
[Wed Apr 04 14:59:55.408060 2018] [mpm_event:error] [pid 21247:tid 
140439370192768] AH00485: scoreboard is full, not at MaxRequestWorkers


This does not appear to be disk-space related; see the cacti graphs
for