On 30/01/15 19:06, Mike Bayer wrote:
Matthew Booth mbo...@redhat.com wrote:
At some point in the near future, hopefully early in L, we're intending
to update Nova to use the new database transaction management in
oslo.db's enginefacade.
Spec:
http://git.openstack.org/cgit/openstack/oslo-specs/plain/specs/kilo/make-enginefacade-a-facade.rst
Implementation:
https://review.openstack.org/#/c/138215/
One of the effects of this is that we will always know when we are in a
read-only transaction, or a transaction which includes writes. We intend
to use this new contextual information to make greater use of read-only
slave databases. We are currently proposing that if an admin has
configured a slave database, we will use the slave for *all* read-only
transactions. This would make the use_slave parameter passed to some
Nova apis redundant, as we would always use the slave where the context
allows.
However, using a slave database has a potential pitfall when mixed with
separate write transactions. A caller might currently:
1. start a write transaction
2. update the database
3. commit the transaction
4. start a read transaction
5. read from the database
The client might expect data written in step 2 to be reflected in data
read in step 5. I can think of 3 cases here:
1. A short-lived RPC call is using multiple transactions
This is a bug which the new enginefacade will help us eliminate. We
should not be using multiple transactions in this case. If the reads are
in the same transaction as the write: they will be on the master, they
will be consistent, and there is no problem. As a bonus, lots of these
will be race conditions, and we'll fix at least some.
2. A long-lived task is using multiple transactions between long-running
sub-tasks
In this case, for example creating a new instance, we genuinely want
multiple transactions: we don't want to hold a database transaction open
while we copy images around. However, I can't immediately think of a
situation where we'd write data, then subsequently want to read it back
from the db in a read-only transaction. I think we will typically be
updating state, meaning it's going to be a succession of write transactions.
3. Separate RPC calls from a remote client
This seems potentially problematic to me. A client makes an RPC call to
create a new object. The client subsequently tries to retrieve the
created object, and gets a 404.
Summary: 1 is a class of bugs which we should be able to find fairly
mechanically through unit testing. 2 probably isn't a problem in
practise? 3 seems like a problem, unless consumers of cloud services are
supposed to expect that sort of thing.
I understand that slave databases can occasionally get very behind. How
behind is this in practise?
How do we use use_slave currently? Why do we need a use_slave parameter
passed in via rpc, when it should be apparent to the developer whether a
particular task is safe for out-of-date data.
Any chance they have some kind of barrier mechanism? e.g. block until
the current state contains transaction X.
General comments on the usefulness of slave databases, and the
desirability of making maximum use of them?
keep in mind that the big win we get from writer()/ reader() is that
writer() can remain pointing to one node in a Galera cluster, and
reader() can point to the cluster as a whole. reader() by default should
definitely refer to the cluster as a whole, that is, “use slave”.
As for issue #3, galera cluster is synchronous replication. Slaves
don’t get “behind” at all. So to the degree that we need to
transparently support some other kind of master/slave where slaves do
get behind, perhaps there would be a reader(synchronous_required=True)
kind of thing; based on configuration, it would be known that
“synchronous” either means we don’t care (using galera) or that we
should use the writer (an asynchronous replication scheme).
This sounds like the crux of the matter to me. After some (admittedly
cursory) reading, it seems that galera can use both synchronous and
asynchronous replication. Up until Friday I had only ever considered
synchronous replication, which would not be a problem.
I think opportunistically using synchronous slaves whenever possible
could only be a win. Are there any unpleasant practicalities which might
mean this isn't the case?
However, it sounds to me like there is at least some OpenStack
deployment in production using asynchronous slaves, otherwise the issue
of 'getting behind' wouldn't have come up. We need to understand:
* Are people actually using asynchronous slaves?
* If so, why did they choose to do that, and
* what are they using them for?
All of this points to the fact that I really don’t think the
directives / flags should say anything about which specific database to
use; using a “slave” or not due to various concerns is dependent on
backend implementation and configuration. The purpose of
