I've been thinking about updating our graph database, in particular the
problems caused by deletion, and how to handle them. In particular,
there are times when you need to figure out what all to delete, and it's
not immediately obvious what that might be...
Here is what a typical NIC and IP address look like in our database:
System servidor
|
|
NIC eth0
|
|
|
IP 1.2.3.4
If I remove the NIC, then I need to remove the IP address - unless...
Unless it has a process listening on that IP/port combination
System servidor
|
|
NIC eth0 TCP service sshd
| |
| IP:port 1.2.3.4:22
| |
| +-----+
| |
| |
IP 1.2.3.4
A further complication can occur when someone is using that service
System servidor
|
|
NIC eth0 TCP service sshd TCP client ssh
| | |
| V |
| IP:port 1.2.3.4:22 <-------+
| |
| +-----+
| |
V V
IP 1.2.3.4
Talking more procedurally, we can discover (and add) an IP address in
one of several contexts:
* We discover a service provider that's using it
* We discover it configured to a NIC
* We discover a client which is talking to it
* We discover it through ARP discovery
If any of these future discovery actions no longer finds the things it
may have added, then they need to be deleted - unless one of the other
inserting relationships still needs it.
This can lead to some circumstances that don't seem to make sense at
first because the various discovery agents aren't all run at the same
time. For example, if there is no NIC, then offering a service that
uses an IP that doesn't seem to use any NIC seems odd - but it can
happen because we may not have discovered that that service is no longer
running - or has migrated to another machine.
The kicker here is that the discovery systems only know what is here
now, they have no idea what they added in the past. Of course, this is
stored in the graph database, but so is a lot of stuff ;-). We could
build special-case code into each of them that knows the things they
might have put in before, and does some form of tree-structured diff to
see what they should delete. This sounds messy and complicated with lots
of complexity in the discovery modules (which I want other people to write)
What we need here in some sense is similar to a reference count system.
Of course, Neo4j is well aware of all the references to a node - but not
all references are equal. What I think we need here is a sort of
ownership system - so that a node disappears if all its owners
(inserters) goes away. Looking at the last diagram, 1.2.3.4 is "owned"
by the NIC, the sshd service, and the ssh client. Once they're all
gone, then it should be deleted as well. The 'TCP service sshd' and the
"TCP client" "own" the IP:port node as well.
If you delete a node, you can follow all its "owning" links and delete
anything which doesn't have any additional owning links.
If we have a single link type for this kind of owning relationship, then
this can be easily done as part of committing the transaction - all in
one place. This should simplify the discovery plugins that populate the
graph database, since adding these "owner" links at insertion time is
much simpler than figuring everything out at deletion time. Since I'm
writing that code (once), then this simplifies the discovery plugins
that other people have to write.
The discovery plugins still do have to do a top level diff - to see if
anything they added in the past needs to be deleted, but it seems like
the ownership idea will make deleting things simpler - and isolate all
the different reasons to insert it from each other.
--
Alan Robertson <[email protected]> - @OSSAlanR
"Openness is the foundation and preservative of friendship... Let me claim
from you at all times your undisguised opinions." - William Wilberforce
_______________________________________________
Assimilation mailing list - Discovery-Driven Monitoring
[email protected]
http://lists.community.tummy.com/cgi-bin/mailman/listinfo/assimilation
http://assimmon.org/
