Hi all,
Right now, there is a need for transactions that span both the network
(updating the rings) and also the database. In other words, I need to
update the database and the network at the same time - and make sure
both parts get completely done.
The REST interface I use to talk to the database makes every request
into a transaction, which slows it down.
If I crash in the middle of an update, I want that update (database and
network) to complete as soon as it can.
What we need is a method of bundling the database and network updates in
a single transaction and ensuring they don't get lost.
In addition, given that a crash can happen at any point trying to apply
this transaction, and the network elements don't understand
transactions, it's necessary for our transactions to be idempotent.
That is, we should be able to apply the same transaction several times
in a row, and get the same result as though we only applied it once.
Fortunately, because of some good planning, and a little dumb luck, our
updates in the network are idempotent - asking it to send heartbeats
twice is the same as asking it to send them once.
Given that we have lots of code that uses and supports JSON, what makes
sense to me is to bundle all the information for a transaction into a
JSON object, then apply that object to the network and the database -
separating it out in two steps. We can then easily persist this JSON
string where ever we like (disk, message queue, etc) so that it doesn't
get lost.
Once this JSON object is persisted somewhere, we can acknowledge the
packet that caused the update - because that update will get done.
Here are the steps involved:
1) A nanoprobe sends a packet that triggers an update
2) Packet is received
3) Updates are computed from the packet
4) Updates are persisted to non-volatile storage
5) Packet from nanoprobe is ACKed
6) Nanoprobe receives ACK (knows to stop resending the packet)
7) Transaction is executed
8) Transaction commitment is recorded (transaction is flushed).
The trick is that the CMA could crash in the middle of or between any
steps. It gets more interesting if the crash happens during or between
steps 4-8. There's also no guarantee that step 6 happens when it is
shown, or if the packet is lost before a crash, that it happens at all
(until a reboot).
Until step 6 happens, the nanoprobe will periodically resend the
packet. If we have a crash, and it takes a while for it to reboot (or
fail over in an HA configuration), then it will just keep resending...
In terms of implementation, I plan on doing it in three steps:
1) Implement a transaction object that collects database and network
updates
2) Change the code to use the transaction object, and commit
(execute) it from an in-memory version
of the transaction
3) Change the code to persist the transaction object before trying
to commit it.
Initially this will probably just be on disk as flat files.
If we get real demand for larger systems than we can accommodate
with a single machine,
we might change to a messaging system. Given our design, this
might take a long time before
it becomes a problem.
When I implement the code to commit the transaction object, I'll write
it using the Neo4j batch facility - which is much more efficient than
what I'm doing now - since a batch update happens as a single transaction.
--
Alan Robertson <[email protected]> - @OSSAlanR
"Openness is the foundation and preservative of friendship... Let me claim
from you at all times your undisguised opinions." - William Wilberforce
_______________________________________________
Assimilation mailing list - Discovery-Driven Monitoring
[email protected]
http://lists.community.tummy.com/cgi-bin/mailman/listinfo/assimilation
http://assimmon.org/
