Participants
============
1. guilhem
2. brett
3. Emiliano
4. Marco
Agenda
======
* New participant: Marco
+ TDF member
+ Authored the LOOL deployment guide on the wiki
+ python dev + sysadmin
+ lurking here for now :-)
* Presentation round
+ guilhem: TDF staff member in charge of infrastructure, main contact for
infra
+ Emiliano: sysadmin, BoD member, part of the oversight area for infra
+ Brett: sysadmin/devops, TDF member, part of the infra team since a few
years; helped with monitoring and metrics collection among other things
+ cloph [absent]: mainly active on dev-related areas like tinderboxes,
weblate and silverstripe
* infratools (GitLab) deprecation
+ https://git.libreoffice.org/infra/salt waiting to be opened
+ deadline to bring down https://infratools.documentfoundation.org: June 30
(Jessie EOL)
+ Brett: want to export the issues, interested to work back on that at some
point
+ tickets could be moved to Redmine (~10 at most)
- AI guilhem: create account for Brett (done)
+ Brett: CI won't be functional anymore [g. it wasn't really], could replace
it with Jenkins if desired
- either stick to a docker-based deployment, or switch to VMs
* OS upgrades:
+ 3 hypervisors upgraded to Buster
+ most guests as well
+ prod guests that are still pending: website, gerrit (waiting for upgrade to
3.1), jenkins
* libvirt API for guest healthchecks (FS %free, running kernel, etc)
+ guilhem: anyone with experience with golang/libvirt API?
- Brett/guilhem: only basic patching, no real dev
+ EV: does the virt stack use plain qemu/KVM or is it behind proxmox or
something else?
- guilhem: KVM, qemu with gfapi block devices and libvirt for the
management layer (glueing/monitoring/multiplexing)
+ Prometheus exporter https://github.com/rumanzo/libvirt_exporter_improved ,
- removes the need to deploy node exports on each guest
- metrics name not that stable accross node-exporter versions
- not all vms are connected to the vpn, would need to rewire or deploy a
PKI
- creates overhead
+ Brett: prometheus has the ability to push metrics too, advantages: no need
to open a port or to connect guests to the vpn
- guilhem: would still need a PKI for client auth though
+ libvirt stats API: network usage, %CPU, %mem, block device metrics (R/W,
flush time and #requests)
- qemu guest agent: https://wiki.qemu.org/Features/GuestAgent can access
more fine-grained info like FS usage, #users, running kernel
. guilhem: would be nice to have dashboards showing how many VMs have a
FS at ≥%85 full, or that run an old kernel,… etc. and configure the
alert system accordingly
.
https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainGetGuestInfo
(requires libvirt ≥5.7.0, guilhem: backported libvirt for that)
. guilhem: wasn't able to patch libvirt_exporter_improved to use that API
call
- Example using the virsh(1) binary; would like to have that data in
/metrics
$ virsh guestinfo vm150 | grep -F -eos.{pretty-name,kernel-release}
-e{total,used}-bytes
os.pretty-name : Debian GNU/Linux 10 (buster)
os.kernel-release : 4.19.0-9-amd64
fs.0.total-bytes : 46909095936
fs.0.used-bytes : 27519135744
- Brett: would be happy to have a look; EV: also interested to help, not
sure about time will allow though
* gerrit upgrade to 3.1
+ before or after 7.0? (pending on cloph's input)
+ git wire protocol v2, gives ~95% perf improvement on up to date trees (from
20MiB to ~100kiB, also faster since git doesn't have to traverse as much)
- might help TBs that are slow to fetch the repo and timeout
* tb69 "Disconnected by cloph : out of diskspace"
+ Do tinderbox owners need to perform routine maintenance?
- g. not FS cleanups and baseline upgrades at least (unless physical login
is needed)
+ Brett: Will install node exporter ≥0.16 on tb69 for monitoring
* Update on https://extensions.libreoffice.org (Q: cloph)
+ user feedback/workflow?
+ vm179 (old plone site) deprecation, deadline June 30 (Jessie EOL)
+ AI cloph: https://extensions.libreoffice.org/en/extensions/show/$NAME calls
to Google (reCAPTCHA), put the comment/rating stuff on a separate page (or
activate it with a button), perhaps even let authenticated users through
- Marco: know a reCAPTCHA alternative that can be self-hosted
. https://github.com/Gregwar/Captcha
. does it work with
https://github.com/silverstripe/silverstripe-spamprotection
(used to protect the page right now)?
* Marco: what is used for metric collections of website visitors?
+ Piwik/Matomo: https://piwik.documentfoundation.org
+ We use it for ~75/80 websites, and also for download and update check
metrics collection (using custom exporters)
+ Fairly large instance with ~6M hits/day
* [rdm#3097] Try out MirrorBits as replacement for MirrorBrain
+ https://redmine.documentfoundation.org/issues/3097
+ Marco: will look at this
- /join #tdf-infra on freenode
* Brett: how did the [Matrix] move go?
+ guilhem: adopted by BoD and staff right now
+ doesn't use Single Sign-On ATM, reverted last year to give clients (eg
mobile) more time to implement SSO authentication
+ EV: Let's brainstom about it and see in the future if it's feasible, maybe
via ML and discuss next/the other infra meeting
+ Brett: was assuming that #tdf-infra would move to [Matrix] and be
discontinued/bridged
. was hopping to connect own server
. making the instance public could maybe lure Telegram users out :-)
* Next call: Tue Jul 21 16:30 UTC 2020
--
Guilhem.
--
To unsubscribe e-mail to: [email protected]
Problems? https://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: https://wiki.documentfoundation.org/Netiquette
List archive: https://listarchives.libreoffice.org/global/website/
Privacy Policy: https://www.documentfoundation.org/privacy
