All +1 I am in favour of Sam's proposal since it mirrors the way most people think about networking. In layman's terms one expects to see a management or control plane, one or more private backplane(s) and finally public facing endpoints.
Adopting a similar convention for Brooklyn that allows us to explicitly address VMs from a management POV seems eminently reasonable. How this is resolved may vary of course but IMO it is the right level to think about this and makes it easier for someone new to Brooklyn to get up to speed. HTH. Best Duncan Johnston-Watt CEO | Cloudsoft Corporation Twitter | @duncanjw Mobile | +44 777 190 2653 Skype | duncan_johnstonwatt Linkedin | www.linkedin.com/in/duncanjohnstonwatt On 8 December 2016 at 14:05, Sam Corbett <[email protected]> wrote: > Thanks for your thoughts Alex. > > The first half more or less makes sense. It would be helpful to me if you > clarified a couple of points. > > I don't understand what you mean when you write that > `management.host.address` assumes a general management.ssh.endpoint and > that this is something we should not do. Why should we not assume this, and > (ignoring naming and my disregard of ports) how does > `brooklyn.ssh.endpoint` differ conceptually? > > I also didn't follow where the name "firewall1" comes from, or how an > enricher on a location works. > > I prefer the idea of publishing "management.network" and > "management.endpoint" to "management.ssh.network" or > "brooklyn.ssh.network". I don't see how you would use the latter form > further down the line, for example when configuring entity sensor feeds. > Maybe this one doesn't matter so much; the proposal you linked says: > > To summarise, sensors can be qualified — e.g. with suffix .public — to > indicate a specific interface. All unqualified sensors including > host.address and main.uri now refer to a “default subnet”, typically > (nowadays) not the public internet. > > I don't think your comments in this thread touch upon this idea but for > simplicity's sake right now I prefer having things continue to use > host.address etc. than thinking in terms of services. > > I'm in rough agreement with the second half of your email, beginning > "Other points to address," but I think a location customiser is the wrong > place to do this. We could make it work but it would be a misuse of an > existing concept. I would prefer to create a separate class. > > Things I intend to do then: > > * test credentials when determining reachability. > > * write LocationNetworkInfoCustomizer (or whatever we choose to call it), > reusing much of the existing code in JcloudsLocation. Its job is to: > > - expose a way of indicating that public or private addresses should be > preferred and default it to private, per the portion of the proposal I > mentioned. > - choose a reachable endpoint for management > - set `host.address` and `host.name` on an entity to this endpoint > (removing the responsibility from MachineLifecycleEffectorTasks) > - publish `host.address.network` and `host.name.network` sensors on > entities for each public and private network jclouds tells us about. > > * not change how feeds choose which address to poll or how > BrooklynAccessUtils works (i.e. they should continue to use host.address). > We can think about management services later. > > I'll try to tidy JcloudsLocation up while I go. > > Any further comments? > > Sam > > > > On 07/12/2016 15:44, Alex Heneveld wrote: > >> Hi Sam, >> >> How does this relate to the strategy suggested in the Networking Proposal >> [1] ? >> >> TL;DR I agree with the intention but think some tweaks to the mechanism >> would make for even better clarify and consistency going forwarD >> >> >> The proposal [1] suggests several things. One is a format for network >> addresses: *host.address.network* >> For instance: >> >> host.address.public: 123.0.0.45 >> host.address.private1: 10.0.0.45 >> host.address.private2: 192.168.0.45 >> >> Another is a format for services: *service.port* . (and *url* and others) >> I've added a suggestion to use the convention . *service.network* . to be >> able to indicate a specific one of the networks above. >> Where required, policies can open up additional ingress or port forwards, >> of the form *service.field.mapped.network* . e.g. >> *http.port.mapped.public* >> . >> >> The format suggested in Sam's proposal -- *management.host.address* -- >> feels inconsistent with the above, and it neglects the need to indicate >> the >> port (e.g. if port forwarding is needed) and bearer (if it's not >> straightforward ssh). I think of management access on a particular >> protocol as a *service* -- eg management or management.ssh or >> *brooklyn.ssh*. >> So the proposal would instead say: >> >> brooklyn.ssh.network: private1 >> brooklyn.ssh.port: 22 >> >> With enrichers then able to create eg *brooklyn.ssh.endpoint: >> 10.0.0.45:22 >> <http://10.0.0.45:22>* >> >> This is saying "the service this entity exposes for Brooklyn to SSH in is >> 10.0.0.45:22". It isn't assuming a general management.ssh endpoint (you >> could, as a different service, but we shouldn't assume that's always the >> case), nor is it assuming a dedicated management network, although again >> it >> could support it (eg replace "private1" with "management" in the above >> network name). >> >> This would let us be consistent with other port-forwarding / enriching >> strategies as well as naming conventions. For instance if Brooklyn needs >> to ssh through port-forwarding (PFW) with say a rule created on >> "firewall1" >> at 10.0.0.1:10001 CIDR'd to Brooklyn we might start with: >> >> ssh.network: private1 >> ssh.port: 22 >> >> and >> >> host.address.firewall1: 10.0.0.1 >> >> enrichers would create this: >> >> ssh.endpoint: 10.0.0.45:22 >> >> but brooklyn wouldn't be able to access it ... instead a PFW customiser >> would set up the following: >> >> ssh.port.mapped.firewall1: 10001 >> ssh.endpoint.mapped.firewall1: 10.0.0.1:10001 >> >> and then brooklyn/location setup would create the following sensors from >> the above: >> >> brooklyn.ssh.network: firewall1 >> brooklyn.ssh.port: 10001 >> * brooklyn.ssh.endpoint: 10.0.0.1:10001 <http://10.0.0.1:10001>* >> >> The bold line above is what Brooklyn will use to make ssh connections -- >> completely unambiguous and it can be populated through different >> strategies. (And if it isn't direct ssh but instead >> ssh-via-an-intermediate-machine, or https, or some other ssh-encoding >> strategy we could also have "brooklyn.ssh.bearer: my-https" ... not >> immediately relevant here but important to keep in mind that not all SSH >> commands are sent via a straightforward ssh.) >> >> >> Other points to address, some of which you touch on, are: >> >> (a) which network is deemed the "default" (used to populate host.address) >> (b) which network Brooklyn uses to connect for ssh purposes >> (c) how are networks named (eg "public", "private1", "private2") >> (d) how do we infer the host name >> (e) which network Brooklyn uses to connect for other monitoring/control >> purposes (http etc) >> >> Currently as you note, Brooklyn when creating a JcloudsSshMachineLocation >> tries to connect to sockets on the reported public addresses and then on >> the reported private addresses, and decides that the first one which >> listens on port 22 is to be used for (a) and (b). It neglects (c) >> altogether and publishes only `host.address` (and `host,name`). >> >> Whilst it is a big ask to make the perfect strategies for all this, I >> think >> a big improvement would be to permit this behaviour to be customised. I >> suggest we provide write a `LocationNetworkInfoCustomizer` instance >> (implementing `LocationCustomizer`) to perform (a)-(d), together with a >> `networkInfoCustomizer` config key to load it (so that we don't disrupt >> other uses of LocationCustomizers). That class could be the default, but >> it could take some additional customization (eg to define specific >> strategies for (b)) and of course a developer could subclass it; this >> allows behaviour to be overridden in either the location's definition or >> in >> an entity's provisioning properties. >> >> The initial behaviour of such an instance could be as follows: >> >> * attempt to find ports 22 to which brooklyn is able to successfully log >> in >> via ssh (not just reach, as you noted Svet is going to fix ... I've had >> bizarre problems in hotels where Brooklyn won't connect because an >> inaccessible 10.x.x.x address of the machine is an address of a machine on >> the hotel wifi!) >> * preferring private addresses >> * but configurable something like "preferPublic" "preferPrivate" >> "allowPublic" etc ... and/or a CIDR preference order >> * give names "public1/2/3" and "private1/2/3" to those networks reported >> by >> jclouds (in future we could support CIDR constraints or subclasses could >> connect to the machine and see which nics they correspond to) >> * do simple things for now like make the network brooklyn uses the default >> and use the current strategies for hostname, but again these could be >> extended in the future >> * publish the sensors above >> >> As for (e) I suggest a similar pattern to explicitly identify the >> brooklyn-accessible endpoints for other services that brooklyn needs >> access >> to, eg creating `brooklyn.http.url` from a `http.port` and `http.url` or >> `http.network`, possibly through an intermediate >> `http.url.mapped.jumphost`. >> >> This should let us solve a lot of issues, not just management network >> conflation but also public/private issues we have when configuring >> clusters >> of nodes who all need to talk to each other. >> >> Best >> Alex >> >> >> [1] >> https://docs.google.com/document/d/1IrWLWunWSl_ScwY3MRICped8 >> eJMjQEH1FbWZJcoK0Iw >> >> On 7 December 2016 at 11:52, Geoff Macartney < >> [email protected]> wrote: >> >> +1 This sounds like a good idea. In most customer deployments I've seen >>> in "previous lives" there has been a separate management network for >>> production deployments, it would be good for Brooklyn to have this as an >>> explicit concept factored out from 'host.address'. >>> >>> Geoff >>> >>> >>> >>> On Wed, 7 Dec 2016 at 11:31 Richard Downer <[email protected]> wrote: >>> >>> +1 >>>> >>>> The host.address and host.subnet.address has always been confusing, at >>>> least for me, and especially so for figuring out what SshMachineLocation >>>> >>> is >>> >>>> going to do. Having a dedicated sensor for the management address with >>>> unambiguous purpose for SshMachineLocation (and others!) seems obvious >>>> to >>>> me! >>>> >>>> Thanks >>>> Richard. >>>> >>>> >>>> On 6 December 2016 at 16:26, Sam Corbett <[email protected] >>>> > >>>> wrote: >>>> >>>> Summary: >>>>> >>>>> Brooklyn conflates the management address of an entity with its public >>>>> address. I want to break this by publishing a new sensor called >>>>> management.host.address on entities that use JcloudsLocation and using >>>>> >>>> it >>> >>>> in preference to host.address when creating SSH connections and polling >>>>> feeds. Its value is the entity's host.subnet.address if that is >>>>> >>>> accessible, >>>> >>>>> otherwise host.address. >>>>> >>>>> Some background: >>>>> >>>>> JcloudsLocation makes a guess at the best host and port to use for SSH >>>>> connections to each instance. The value it chooses subsequently >>>>> >>>> informs a >>> >>>> variety of entity sensors, most importantly the "host.address" sensor >>>>> >>>> which >>>> >>>>> itself informs the address at which various feeds are polled and other >>>>> sensors like "main.uri". Right now Brooklyn always prefers a value from >>>>> >>>> the >>>> >>>>> set of "public" addresses returned by the cloud. Internally to >>>>> JcloudsLocation the value that is picked is referred to as the >>>>> >>>> "management >>>> >>>>> host and port", but by subsequently using it for host.address we >>>>> >>>> conflate >>> >>>> it with the publicly accessible address. >>>>> >>>>> An obvious change to make to this is to have Brooklyn use the private >>>>> address for SSH connections when it's on the same subnet as the thing >>>>> >>>> it >>> >>>> has provisioned. In order to do this without mucking up all of the >>>>> >>>> existing >>>> >>>>> assumptions I propose we introduce a new sensor called >>>>> "management.host.address", whose value is a reachable private address, >>>>> >>>> if >>> >>>> one exists, and otherwise the value chosen for the public address. When >>>>> creating SSH connections SshMachineLocation would check for the >>>>> >>>> management >>>> >>>>> address and fall back to its current behaviour if it's unset. >>>>> >>>>> An alternative is to have SshMachineLocation itself work out whether it >>>>> can connect to a private address. I do not like this option because we >>>>> would be unable to reuse the information that a private address is >>>>> >>>> better >>> >>>> in entity feeds and BrooklynAccessUtils. >>>>> >>>>> Svet raised the excellent point that we risk having an instance's >>>>> >>>> private >>> >>>> address match an irrelevant machine on the same network as Brooklyn. To >>>>> resolve this he suggests that we change the check for reachability to >>>>> >>>> also >>>> >>>>> test credentials rather than simply trying to open a socket as happens >>>>> >>>> at >>> >>>> the moment. >>>>> >>>>> I'm going to start on an implementation of this. Any feedback or >>>>> >>>> questions? >>>> >>>>> Sam >>>>> >>>>> >
