Hi Pete, On 08/14/2013 06:16 PM, [email protected] wrote: > Alan, > > This is a most interesting subject :-) My comments inline...
It is indeed an interesting subject. Thanks for chiming in! > > On Aug 14, 2013, at 11:13 AM, Alan Robertson <[email protected] > <mailto:[email protected]>> wrote: > >> I've been thinking about updating our graph database, in particular >> the problems caused by deletion, and how to handle them. In >> particular, there are times when you need to figure out what all to >> delete, and it's not immediately obvious what that might be... >> >> Here is what a typical NIC and IP address look like in our database: >> >> System servidor >> | >> | >> NIC eth0 >> | >> | >> | >> IP 1.2.3.4 >> >> If I remove the NIC, then I need to remove the IP address - unless... >> Unless it has a process listening on that IP/port combination >> >> >> System servidor >> | >> | >> NIC eth0 TCP service sshd >> | | >> | IP:port 1.2.3.4:22 >> | | >> | +-----+ >> | | >> | | >> IP 1.2.3.4 >> >> A further complication can occur when someone is using that service >> >> System servidor >> | >> | >> NIC eth0 TCP service sshd TCP client ssh >> | | | >> | V | >> | IP:port 1.2.3.4:22 <-------+ >> | | >> | +-----+ >> | | >> V V >> IP 1.2.3.4 >> >> Talking more procedurally, we can discover (and add) an IP address in >> one of several contexts: >> >> * We discover a service provider that's using it >> * We discover it configured to a NIC >> * We discover a client which is talking to it >> * We discover it through ARP discovery >> >> If any of these future discovery actions no longer finds the things >> it may have added, then they need to be deleted - unless one of the >> other inserting relationships still needs it. > > I disagree. Because things happen at their own time scale, and not in any particularly predictable way, if it needs to be cleaned up - it will eventually be cleaned up. Your comments about garbage collection below indicate that you know that things need to be cleaned up. > > What does it mean to "remove the NIC"? I assume it means that in the > physical world, someone has pulled the NIC card out of the machine - > assume for now that the hardware allows this while the machine is > running. Our Cisco Ethernet switches do, and can even be configured > to send an SNMP trap when cards are removed or inserted. Anyway, the > pulled NIC still exists, and has a MAC address burned into it, and a > serial number, manufacturer, number-of-interfaces, etc. The humans > may put the NIC on a shelf, but still expect that it will appear in an > inventory report. I think the best approach is to make the database > model what has happened in the physical world - remove the link > between servidor and the NIC, but don't delete the NIC. When it > reappears later, make a new connection to the existing database > entity. This approach would permit you to keep history records for > each database entity, making it possible to track where a NIC has > been. This is of real value if we're talking about MAC addresses - > our security guy loves the idea of tracking where a rogue MAC has > appeared on our network. If the OS thinks the NIC is gone - it _is_ gone. It's not in doubt in any way. It means someone pulled the card, or unplugged a USB NIC, or disabled it in the BIOS or something similar. What's confusing is the fact that OSes (and BIOSes) love to rename NICs from time to time. Don't yet have a strategy for dealing with that... Keeping a history is an interesting idea and has lots of uses. I've thought about it off and on for a long time. Don't have any answers I like yet (that includes yours - sorry). But it's also true that this is not an inventory system. We don't have any way to track hardware which is on the shelf, or has never been powered on, or similar. Inventory systems are really useful, and we will eventually plug into them - but that's not our purpose. Charles Edward Bedón Cortazar is on the mailing list - and he runs the neotropic open source inventory project. I CCed him to get his attention ;-). Also note that not all NICs are real - some belong to VMs -- another interesting case... > > When you notice that the NIC has been pulled, you may want to "clean > up" by immediately testing the things that depend on the NIC. The CMA > knows that the NIC was attached to an ssh process, so the CMA could > immediately check if the ssh session still exists. If not, the ssh > session could be deleted, because it's a logical entity - it's really > gone for all time. That is costly and will get cleaned up soon enough -- or not -- if the IP moves to a different interface or even a different computer. But if a client process is still trying to connect to it, you don't want to lose that relationship - since they think that you already know it, you don't want to delete it - because they won't resend what didn't change. If there is no such relationship, then it will go away. How soon we notice something depends on how fast we run the discovery "polling" operations in the nanoprobes. The usual tradeoffs apply. But things scale much better the more we can give to the nanoprobes... That's our strategy. The chances that you're going to look before things get corrected is small - and will never be zero. A note about clients: Because client connections may be short-lived, it's necessary to cache them for "a while" - so that we don't think they're continually coming and going - when in fact the dependency exists even if the last time we looked it wasn't active. Some client connections should probably be filtered out to lower the noise level - that would typically include ssh (I know I keep using it as example - it's so easy to set up and demonstrate). > > >> This can lead to some circumstances that don't seem to make sense at >> first because the various discovery agents aren't all run at the same >> time. For example, if there is no NIC, then offering a service that >> uses an IP that doesn't seem to use any NIC seems odd - but it can >> happen because we may not have discovered that that service is no >> longer running - or has migrated to another machine. > > It's not odd. There are IPs that don't have NICs. How about 127.0.0.1? > > The NIC and the IP address are not connected. When you pull the NIC, > the IP address doesn't remain in the NIC. The connection between the > NIC and the IP address in the OS. What's missing in the model is > interfaces. When you pull the NIC out, the interface disappears from > the machine. The IP address is connected to the interface. So: > machines have interfaces. Some interfaces have IP addresses. Some > interfaces have NICs, but some don't, like lo0. > > Loopback interfaces can have IP addresses that can be reached from > outside the machine. Our switches and routers do. If you have a route established, that's certainly true. There are some HA techniques (particularly used with LVSs) that rely on this. Not being a network expert, I never fully understood them the way you no doubt do ;-). > >> The kicker here is that the discovery systems only know what is here >> now, they have no idea what they added in the past. Of course, this >> is stored in the graph database, but so is a lot of stuff ;-). We >> could build special-case code into each of them that knows the things >> they might have put in before, and does some form of tree-structured >> diff to see what they should delete. This sounds messy and >> complicated with lots of complexity in the discovery modules (which I >> want other people to write) > > I don't think it's complex. The discovery systems should tell the CMA > when things have changed. They keep a snapshot of what they've > discovered, and say nothing when nothing has changed. When a change > occurs, they tell the CMA the new state of things via "add" and/or > "delete" notifications. When a nanoprobe has been offline, and comes > back online, the CMA should request a complete report of everything > the nanoprobe knows, and then the CMA should compare it to what's in > the database, and add/delete as needed. The current implementation is just to send back the current information. The design is to avoid any complexity or policy in the agents, and concentrate it in the CMA - where it is easier to fix if it turns out to be incorrect, or the policy changes. It turns out that the CMA has a slightly better idea of history than the nanoprobes do (nanoprobes have no persistent store over reboots). > >> What we need here in some sense is similar to a reference count >> system. Of course, Neo4j is well aware of all the references to a >> node - but not all references are equal. What I think we need here >> is a sort of ownership system - so that a node disappears if all its >> owners (inserters) goes away. Looking at the last diagram, 1.2.3.4 >> is "owned" by the NIC, the sshd service, and the ssh client. Once >> they're all gone, then it should be deleted as well. The 'TCP service >> sshd' and the "TCP client" "own" the IP:port node as well. If you >> delete a node, you can follow all its "owning" links and delete >> anything which doesn't have any additional owning links. > > As I wrote above, I think perhaps you want to lean towards not > deleting things. Just disconnect them. You could mark every database > entity with a timestamp of the last time it was discovered. You could > have a garbage collection process/thread in the CMA that inspects > every entity and deletes it if nothing points to it AND it's been a > long time since it's been discovered. That GC thing can run lazily, > on command, or whatever. GC /is/ deleting. It's a policy decision about when to delete what. This kind of decision belongs in the CMA. It's a classic decision - GC versus reference counting ;-). If you have a large environment, then GC will take a VERY long time - maybe weeks... And in the meantime, you will have a confusing picture of a potentially large part of the world. I tend to shy away from GC for this reason. The question is - what's the use case. The one I'm trying to attack first is understanding the true state of things right now. There is value to understanding the past, and value to set a benchmark for future plans. Those will have to come later. > > Some other things that occur to me: > > Suppose I have a home network that contains a printer. It's been > discovered. I turn it off. I go on a vacation for 3 weeks and while > away, I buy a shiny new device. I come home and want to give the new > device an IP address. The printer is still turned off. I look in my > database to find an unused IP address. Will the database contain the > IP address of my printer, or will it claim the IP address is unused? The way this usually works is that addresses are assigned by DHCP. I'm not trying to replace DHCP - just to cope with it ;-). DHCP is a classic example of why you need to clean things up ;-). But to answer your question: The IP address would not appear in the database unless someone was trying to connect to it. Then it would show up -- but because of the client, not the server (printer). [This is what all the reference count discussion is about]. By the way, my wife's school is a good example of what we ought eventually be able to cope with - it would be stressful :-D. Every hour they power down a few hundred laptops and ipads, and then power them up again the next hour. So, at least half of the inventory likely gets 16 different IP addresses each day ;-). It's a good case to think about... But right now, I'm just trying to get something working for the simpler cases. Eventually there is great value in handling such cases (BYOD comes to mind). But right now, I need to get the database updating to work correctly for more normal cases ;-). > > Suppose I'm discovering a network that contains 2 home networks. Both > networks use NAT, and both networks contain the same IP address. Will > the database handle it properly? Like, if there's an ssh session > inside such a home network, will the database differentiate between > the "duplicate" IP addresses, and make the proper connections? Right now, the code as currently written would not handle that correctly. It's a pretty unusual in a corporate network to have the same IP appear in more than one place - because of the confusion it causes human beings. If that comes up, we will have to figure out how to deal with it. > >> If we have a single link type for this kind of owning relationship, >> then this can be easily done as part of committing the transaction - >> all in one place. This should simplify the discovery plugins that >> populate the graph database, since adding these "owner" links at >> insertion time is much simpler than figuring everything out at >> deletion time. Since I'm writing that code (once), then this >> simplifies the discovery plugins that other people have to write. >> >> The discovery plugins still do have to do a top level diff - to see >> if anything they added in the past needs to be deleted, but it seems >> like the ownership idea will make deleting things simpler - and >> isolate all the different reasons to insert it from each other. > > If it helps to add the ownership links, it sounds good. Perhaps this > general problem is addressed in documentation about Neo4j. I had a long conversation in person with one of their experts while I was at OSCON. Graph databases are pretty new, and everyone is still learning about them -- including their implementers ;-). A lot of their current use cases don't involve a lot of updating - or at least not updating this complex - and the distributed nature of the data collection/creation makes it more complicated... I'll be speaking at one of their conferences in a few weeks, and I'll discuss this with them in person - and may spend a few minutes on it in my talk. I've started a conversation with them about this on their mailing list, but this particular discussion hasn't hit them yet - I wanted to hit this crowd with it first. In summary: You have a wide variety of use cases in mind - and they sound good. I have identified an initial class of use cases that I'm pursuing that I understand as basic - from both the use case perspective and from the technical perspective. Gotta crawl before we walk, and walk before we run... But please continue to think about this and chime in as you have time! -- Alan Robertson <[email protected]> - @OSSAlanR "Openness is the foundation and preservative of friendship... Let me claim from you at all times your undisguised opinions." - William Wilberforce
_______________________________________________ Assimilation mailing list - Discovery-Driven Monitoring [email protected] http://lists.community.tummy.com/cgi-bin/mailman/listinfo/assimilation http://assimmon.org/
