Hi,
Here's my summary of the discussions we had in the Oslo room at the PTG.
Please feel free to reply with any additions if I missed something or
correct anything I've misrepresented.
oslo.config drivers for secret management
-----------------------------------------
The oslo.config implementation is in progress, while the Castellan
driver still needs to be written. We want to land this early in Rocky
as it is a significant change in architecture for oslo.config and we
want it to be well-exercised before release.
There are discussions with the TripleO team around adding support for
this feature to its deployment tooling and there will be a functional
test job for the Castellan driver with Custodia.
There is a weekly meeting in #openstack-meeting-3 on Tuesdays at 1600
UTC for discussion of this feature.
oslo.config driver implementation: https://review.openstack.org/#/c/513844
spec:
https://specs.openstack.org/openstack/oslo-specs/specs/queens/oslo-config-drivers.html
Custodia key management support for Castellan:
https://review.openstack.org/#/c/515190/
"stable" libraries
------------------
Some of the Oslo libraries are in a mature state where there are very
few, if any, meaningful changes to them. With the removal of the
requirements sync process in Rocky, we may need to change the release
process for these libraries. My understanding was that there were no
immediate action items for this, but it was something we need to be
aware of.
dropping support for mox3
-------------------------
There was some concern that no one from the Oslo team is actually in a
position to support mox3 if something were to break (such as happened in
some libraries with Python 3.6). Since there is a community goal to
remove mox from all OpenStack projects in Rocky this will hopefully not
be a long-term problem, but there was some discussion that if projects
needed to keep mox for some reason that they would be asked to provide a
maintainer for mox3. This topic is kind of on hold pending the outcome
of the community goal this cycle.
automatic configuration migration on upgrade
--------------------------------------------
There is a desire for oslo.config to provide a mechanism to
automatically migrate deprecated options to their new location on
version upgrades. This is a fairly complex topic that I can't cover
adequately in a summary email, but there is a spec proposed at
https://review.openstack.org/#/c/520043/ and POC changes at
https://review.openstack.org/#/c/526314/ and
https://review.openstack.org/#/c/526261/
One outcome of the discussion was that in the initial version we would
not try to handle complex migrations, such as the one that happened when
we combined all of the separate rabbit connection opts into a single
connection string. To start with we will just raise a warning to the
user that they need to handle those manually, but a templated or
hook-based method of automating those migrations could be added as a
follow-up if there is sufficient demand.
oslo.messaging plans
--------------------
There was quite a bit discussed under this topic. I'm going to break it
down into sub-topics for clarity.
oslo.messaging heartbeats
=========================
Everyone seemed to be in favor of this feature, so we anticipate
development moving forward in Rocky. There is an initial patch proposed
at https://review.openstack.org/546763
We felt that it should be possible to opt in and out of the feature, and
that the configuration should be done at the application level. This
should _not_ be an operator decision as they do not have the knowledge
to make it sanely.
There was also a desire to have a TTL for messages.
bug cleanup
===========
There are quite a few launchpad bugs open against oslo.messaging that
were reported against old, now unsupported versions. Since we have the
launchpad bug expirer enabled in Oslo the action item proposed for such
bugs was to mark them incomplete and ask the reporter to confirm that
they still occur against a supported version. This way bugs that don't
reproduce or where the reporter has lost interest will eventually be
closed automatically, but bugs that do still exist can be updated with
more current information.
deprecations
============
The Pika driver will be deprecated in Rocky. To our knowledge, no one
has ever used it and there are no known benefits over the existing
Rabbit driver.
Once again, the ZeroMQ driver was proposed for deprecation as well. The
CI jobs for ZMQ have been broken for a while, and there doesn't seem to
be much interest in maintaining them. Furthermore, the breakage seems
to be a fundamental problem with the driver that would require
non-trivial work to fix.
Given that ZMQ has been a consistent pain point in oslo.messaging over
the past few years, it was proposed that if someone does step forward
and want to maintain it going forward then we should split the driver
off into its own library which could then have its own core team and
iterate independently of oslo.messaging. However, at this time the plan
is to propose the deprecation and start that discussion first.
CI
==
Need to migrate oslo.messaging to zuulv3 native jobs. The
openstackclient library was proposed as a good example of how to do so.
We also want to have voting hybrid messaging jobs (where the
notification and rpc messages are sent via different backends). We will
define a devstack job variant that other projects can turn on if desired.
We also want to add amqp1 support to pifpaf for functional testing.
Low level messaging API
=======================
A proposal for a new oslo.messaging API to expose lower level messaging
functionality was proposed. There is a presentation at
https://docs.google.com/presentation/d/1mCOGwROmpJvsBgCTFKo4PnK6s8DkDVCp1qnRnoKL_Yo/edit?usp=sharing
This seemed to generally be well-received by the room, and dragonflow
and neutron reviewers were suggested for the spec.
Kafka
=====
Andy Smith gave an update on the status of the Kafka driver. Currently
it is still experimental, and intended to be used for notifications
only. There is a presentation with more details in
https://docs.google.com/presentation/d/e/2PACX-1vQpaSSm7Amk9q4sBEAUi_IpyJ4l07qd3t5T_BPZkdLWfYbtSpSmF7obSB1qRGA65wjiiq2Sb7H2ylJo/pub?start=false&loop=false&delayms=3000&slide=id.p
testing for Edge/FEMDC use cases
================================
Matthieu Simonin gave a presentation about the testing he has done
related to messaging in the Edge/FEMDC scenario where messaging targets
might be widely distributed. The slides can be found at
https://docs.google.com/presentation/d/1LcF8WcihRDOGmOPIU1aUlkFd1XkHXEnaxIoLmRN4iXw/edit#slide=id.p3
In short, there is a desire to build clouds that have widely distributed
nodes such that content can be delivered to users from a location as
close as possible. This puts a lot of pressure on the messaging layer
as compute nodes (for example) could be halfway around the world from
the control nodes, which is problematic for a broker-based system such
as Rabbit. There is some very interesting data comparing Rabbit with a
more distributed AMQP1 system based on qpid-dispatch-router. In short,
the distributed system performed much better for this use case, although
there was still some concern raised about the memory usage on the client
side with both drivers. Some followup is needed on the oslo.messaging
side to make sure we aren't leaking/wasting resources in some messaging
scenarios.
For further details I suggest taking a look at the presentation.
mutable configuration
---------------------
This is also a community goal for Rocky, and Chang Bo is driving its
adoption. There was some discussion of how to test it, and also that we
should provide an example of turning on mutability for the debug option
since that is the target of the community goal. The cinder patch can be
found here: https://review.openstack.org/#/c/464028/ Turns out it's
really simple!
Nova is also using this functionality for more complex options related
to upgrades, so that would be a good place to look for more advanced use
cases.
Full documentation for the mutable config options is at
https://docs.openstack.org/oslo.config/latest/reference/mutable.html
The goal status is being tracked in
https://storyboard.openstack.org/#!/story/2001545
Chang Bo was also going to talk to Lance about possibly coming up with a
burndown chart like the one he had for the policy in code work.
oslo healthcheck middleware
---------------------------
As this ended up being the only major topic for the afternoon, the
session was unfortunately lightly attended. However, the self-healing
SIG was talking about related topics at the same time so we ended up
moving to that room and had a good discussion.
Overall the feature seemed to be well-received. There is some security
concern with exposing service information over an un-authenticated
endpoint, but because there is no authentication supported by the health
checking functionality in things like Kubernetes or HAProxy this is
unavoidable. The feature won't be mandatory, so if this exposure is
unacceptable it can be turned off (with a corresponding loss of
functionality, of course).
There was also some discussion of dropping the asynchronous nature of
the checks in the initial version in order to keep the complexity to a
minimum. Asynchronous testing can always be added later if it proves
necessary.
The full spec is at https://review.openstack.org/#/c/531456
oslo.config strict validation
-----------------------------
I actually had discussions with multiple people about this during the
week. In both cases, they were just looking for a minimal amount of
validation that would catch an error such at "devug=True". Such a
validation might be fairly simple to write now that we have the
YAML-based sample config with (ideally) information about all the
options available to set in a project. It should be possible to compare
the options set in the config file with the ones listed in the sample
config and raise warnings for any that don't exist.
There is also a more complete validation spec at
http://specs.openstack.org/openstack/oslo-specs/specs/ocata/oslo-validator.html
and a patch proposed at https://review.openstack.org/#/c/384559/
Unfortunately there has been little movement on that as of late, so it
might be worthwhile to implement something more minimalist initially and
then build from there. The existing patch is quite significant and
difficult to review.
Conclusion
----------
I feel like there were a lot of good discussions at the PTG and we have
plenty of work to keep the small Oslo team busy for the Rocky cycle. :-)
Thanks to everyone who participated and I look forward to seeing how
much progress we've made at the next Summit and PTG.
-Ben
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
