Alan, This is a most interesting subject :-) My comments inline…
On Aug 14, 2013, at 11:13 AM, Alan Robertson <[email protected]> wrote: > I've been thinking about updating our graph database, in particular the > problems caused by deletion, and how to handle them. In particular, there > are times when you need to figure out what all to delete, and it's not > immediately obvious what that might be... > > Here is what a typical NIC and IP address look like in our database: > > System servidor > | > | > NIC eth0 > | > | > | > IP 1.2.3.4 > > If I remove the NIC, then I need to remove the IP address - unless... Unless > it has a process listening on that IP/port combination > > > System servidor > | > | > NIC eth0 TCP service sshd > | | > | IP:port 1.2.3.4:22 > | | > | +-----+ > | | > | | > IP 1.2.3.4 > > A further complication can occur when someone is using that service > > System servidor > | > | > NIC eth0 TCP service sshd TCP client ssh > | | | > | V | > | IP:port 1.2.3.4:22 <-------+ > | | > | +-----+ > | | > V V > IP 1.2.3.4 > > Talking more procedurally, we can discover (and add) an IP address in one of > several contexts: > We discover a service provider that's using it > We discover it configured to a NIC > We discover a client which is talking to it > We discover it through ARP discovery > If any of these future discovery actions no longer finds the things it may > have added, then they need to be deleted - unless one of the other inserting > relationships still needs it. I disagree. What does it mean to "remove the NIC"? I assume it means that in the physical world, someone has pulled the NIC card out of the machine - assume for now that the hardware allows this while the machine is running. Our Cisco Ethernet switches do, and can even be configured to send an SNMP trap when cards are removed or inserted. Anyway, the pulled NIC still exists, and has a MAC address burned into it, and a serial number, manufacturer, number-of-interfaces, etc. The humans may put the NIC on a shelf, but still expect that it will appear in an inventory report. I think the best approach is to make the database model what has happened in the physical world - remove the link between servidor and the NIC, but don't delete the NIC. When it reappears later, make a new connection to the existing database entity. This approach would permit you to keep history records for each database entity, making it possible to track where a NIC has been. This is of real value if we're talking about MAC addresses - our security guy loves the idea of tracking where a rogue MAC has appeared on our network. When you notice that the NIC has been pulled, you may want to "clean up" by immediately testing the things that depend on the NIC. The CMA knows that the NIC was attached to an ssh process, so the CMA could immediately check if the ssh session still exists. If not, the ssh session could be deleted, because it's a logical entity - it's really gone for all time. > This can lead to some circumstances that don't seem to make sense at first > because the various discovery agents aren't all run at the same time. For > example, if there is no NIC, then offering a service that uses an IP that > doesn't seem to use any NIC seems odd - but it can happen because we may not > have discovered that that service is no longer running - or has migrated to > another machine. It's not odd. There are IPs that don't have NICs. How about 127.0.0.1? The NIC and the IP address are not connected. When you pull the NIC, the IP address doesn't remain in the NIC. The connection between the NIC and the IP address in the OS. What's missing in the model is interfaces. When you pull the NIC out, the interface disappears from the machine. The IP address is connected to the interface. So: machines have interfaces. Some interfaces have IP addresses. Some interfaces have NICs, but some don't, like lo0. Loopback interfaces can have IP addresses that can be reached from outside the machine. Our switches and routers do. > The kicker here is that the discovery systems only know what is here now, > they have no idea what they added in the past. Of course, this is stored in > the graph database, but so is a lot of stuff ;-). We could build special-case > code into each of them that knows the things they might have put in before, > and does some form of tree-structured diff to see what they should delete. > This sounds messy and complicated with lots of complexity in the discovery > modules (which I want other people to write) I don't think it's complex. The discovery systems should tell the CMA when things have changed. They keep a snapshot of what they've discovered, and say nothing when nothing has changed. When a change occurs, they tell the CMA the new state of things via "add" and/or "delete" notifications. When a nanoprobe has been offline, and comes back online, the CMA should request a complete report of everything the nanoprobe knows, and then the CMA should compare it to what's in the database, and add/delete as needed. > What we need here in some sense is similar to a reference count system. Of > course, Neo4j is well aware of all the references to a node - but not all > references are equal. What I think we need here is a sort of ownership > system - so that a node disappears if all its owners (inserters) goes away. > Looking at the last diagram, 1.2.3.4 is "owned" by the NIC, the sshd service, > and the ssh client. Once they're all gone, then it should be deleted as > well. The 'TCP service sshd' and the "TCP client" "own" the IP:port node as > well. If you delete a node, you can follow all its "owning" links and delete > anything which doesn't have any additional owning links. As I wrote above, I think perhaps you want to lean towards not deleting things. Just disconnect them. You could mark every database entity with a timestamp of the last time it was discovered. You could have a garbage collection process/thread in the CMA that inspects every entity and deletes it if nothing points to it AND it's been a long time since it's been discovered. That GC thing can run lazily, on command, or whatever. Some other things that occur to me: Suppose I have a home network that contains a printer. It's been discovered. I turn it off. I go on a vacation for 3 weeks and while away, I buy a shiny new device. I come home and want to give the new device an IP address. The printer is still turned off. I look in my database to find an unused IP address. Will the database contain the IP address of my printer, or will it claim the IP address is unused? Suppose I'm discovering a network that contains 2 home networks. Both networks use NAT, and both networks contain the same IP address. Will the database handle it properly? Like, if there's an ssh session inside such a home network, will the database differentiate between the "duplicate" IP addresses, and make the proper connections? > If we have a single link type for this kind of owning relationship, then this > can be easily done as part of committing the transaction - all in one place. > This should simplify the discovery plugins that populate the graph database, > since adding these "owner" links at insertion time is much simpler than > figuring everything out at deletion time. Since I'm writing that code > (once), then this simplifies the discovery plugins that other people have to > write. > > The discovery plugins still do have to do a top level diff - to see if > anything they added in the past needs to be deleted, but it seems like the > ownership idea will make deleting things simpler - and isolate all the > different reasons to insert it from each other. If it helps to add the ownership links, it sounds good. Perhaps this general problem is addressed in documentation about Neo4j. -- Pete
_______________________________________________ Assimilation mailing list - Discovery-Driven Monitoring [email protected] http://lists.community.tummy.com/cgi-bin/mailman/listinfo/assimilation http://assimmon.org/
