Hi - Great suggestions, Dan.
To recap, we followed that up with a few other ideas on irc and we eventually came to a point to test some of this, with slight modification. UI also ships with a configuration file that can override the endpoint information received from Keystone. The file is located at /var/www/openstack-tripleo-ui/dist/tripleo_ui_config.js. Part of making this work means enabling SSL on the Undercloud when the UI component is selected for installation in undercloud.conf. I think this is going to be a pretty reasonable request, but I'm interested in hearing feedback from this, and what other implications it may have, that I can't think of. The changes I made were all one-off, unmanaged changes, just to test this idea out. I'll be doing some more tests but will probably be looking for acceptance shortly. Once SSL was enabled on the Undercloud, I made two edits to haproxy.cfg that were pretty straightforward: added a 'listen' server directive for UI to both terminate SSL and forward the request to Apache, and added a 'bind' statement for each service that UI expects to talk to (keystone, heat, ironic, mistral, swift, zaqar-websocket). Once those configuration changes were made, I had a very pleasant experience using the UI. It worked exactly as expected. I think this might be a viable option. Thoughts? Thanks! -dant On Fri, Sep 30, 2016 at 12:21 PM, Dan Sneddon <dsned...@redhat.com> wrote: > Thinking about this a little more, creating a new unified endpoint on the > same port as the UI doesn't solve the problem at hand. The UI will use the > service catalog to find endpoints, so we would need to change the endpoints > in the service catalog, which changes the flow for the underlying services > as well. > > Simply moving the control plane services to the external network won't > work well in environments where the control plane is isolated and > non-routed. The Undercloud can forward packets, but then becomes a > bottleneck and a SPOF. > > A few approaches come to mind, but none of these are quick fixes: > > * Change the UI to get its list of endpoints from somewhere other than the > service catalog and customize this with URLs that point to the Public VIP. > Duplicate the services required for the UI on both control plane and > external network. This might make it possible to make all connections over > port 443, which is more firewall-friendly (which would be desirable or not > depending on what kind of firewalling and traffic management is wanted). > > * Relax the rp_filter settings on the Controllers so they accept packets > destined for the external network on their control plane interfaces; add a > static route to the Public VIP via the control plane VIP on all > non-controller nodes. Modify the service catalog to point to the public VIP > for the services the UI needs. This would need to be combined with a > security review to determine if additional iptables rules are required. > > * Split the service catalog, so we have an internal and an external view > depending on where the query came from. I'm not sure how feasible this is. > > Of these, I think the rp_filter settings are the only ones that could be > done solely with TripleO code changes. That might be worth investigating. > > Dan Sneddon | Principal OpenStack Engineer | dsned...@redhat.com > > > On Sep 30, 2016, at 11:36 AM, Dan Sneddon <dsned...@redhat.com> wrote: > > I don't think we can rely on the Undercloud as an API proxy unless we > address the lack of HA on the Undercloud. > > Wouldn't this be better implemented as as a single, name-based HAProxy > instance running SSL on port 443 on the overcloud Public VIP? Then we could > have the same endpoint for Horizon and every other API. > > I actually implemented this scheme in Havana before I joined Red Hat. At > the time, we had to have a complex HAProxy config and patch the end points > to support name-based URLs. I think some work has been done in OpenStack > now to support this model, but I'm not sure where it stands. > > Dan Sneddon | Principal OpenStack Engineer | dsned...@redhat.com > > > On Sep 30, 2016, at 9:44 AM, Dan Trainor <dtrai...@redhat.com> wrote: > > Hi - > > I re-read your email a few times and like a few of the things that I see, > but I'd love some more clarification. As I read it, a few of these things > conflict. I believe you're suggesting that we don't make these services > listen on a public interface due to security concerns (and of course, > enabling SSL would break this because haproxy would listen on these > interfaces/ports), but this approach would be acceptable if HAProxy was > listening on these ports, terminating SSL, and sending them to each > respective service backend. Am I correct i understanding this? > > Are you suggesting that these endpoint ports would still be externally > accessible on the primary (public) interface of the Undercloud, but just > managed by HAProxy? I think that's an acceptable approach. Even if these > endpoints are, like you suggested, listening on a separate network or IP as > the Undercloud's primary interface, at least then it would be easier for > organizations to enforce network access policies to these ports, and > subsequently, these services that UI needs to talk to directly. > > I'm also perfectly fine with suggesting that if UI is installed, then this > forces the Undercloud to be SSL enabled. This would be a good way to move > the idea of a secured, by default SSL-enabled Undercloud forward a little > more, which is something we'd definitely like to see more. > > Thoughts? > > Thanks > -dant > > > > On Thu, Sep 29, 2016 at 9:01 AM, Dan Trainor <dtrai...@redhat.com> wrote: > >> Hi, Juan - >> >> Actually, the third option is also not an option in the current >>> undercloud setup, since making the services listen in 0.0.0.0 will break >>> HAProxy. So when you're deploying with TLS things will break since we use >>> HAProxy to terminate TLS connections. >>> >> >> Ah, that's correct, isn't it. >> >> >> >>> On the other hand, we also don't want services to listen on 0.0.0.0 >>> since that would become a security concern. We should instead be blocking >>> everything we don't need to have exposed (as we've done with the >>> undercloud's database and rabbitmq). >>> >> >> I don't disagree that we want to focus on least privilege, but we do have >> documentation that talks about how to deal with this. I addressed this in >> my previous message, even if only to illustrate my understanding that there >> would be concerns around this. See more comments about this down below... >> >> >>> >>> Now, as we're trying to move to have more convergence between the >>> undercloud and the overcloud (trying to deploy the undercloud via heat >>> also, as Dan Prince has mentioned), I think some aspecs of this will bring >>> a solution to this problem. for instance, just like we already do in the >>> overcloud, we could have the undercloud's HAProxy always terminate the >>> endpoints, which I'm attempting with these two patches: >>> https://review.openstack.org/#/c/360366 https://review.openstack.org/# >>> /c/360368 . Furthermore, we could have the public endpoints in HAProxy >>> listen on a separate network that's accessible externally, also as we do in >>> the overcloud. That way we don't need to change the actual interfaces the >>> services are listening on, and rely on HAProxy, getting closer to how we do >>> things in the overcloud. It seems to me that it would help solve the >>> problem. >>> >> >> I like that idea. Though, this effectively shifts the process of having >> these services themselves listen on different IPs/ports and offloads that >> to HAProxy. Whatever security concerns we have with opening up network >> communications would still exist, but I think that would be more broadly >> accepted considering these connections are now managed by HAProxy which >> *only* listens for SSL connections. >> >> Another challenge with isolating this traffic on a separate network is >> that we introduce a new dependency that's going to take administrative time >> to set up and configure outside of OpenStack - do we really want that? >> >> Thanks! >> -dant >> >> >> >> >>> >>> On Wed, Sep 28, 2016 at 7:24 PM, Dan Trainor <dtrai...@redhat.com> >>> wrote: >>> >>>> Hi - >>>> >>>> I want to bring up a subject that needs a little more attention. There >>>> are a few ideas floating around but it's important that we get this done >>>> right. >>>> >>>> UI is unique in the sense that it operates almost entirely in a >>>> browser, talking directly to service API endpoints which it either figures >>>> out from they Keystone service catalog as using the publicURL endpoint for >>>> each service, or by specifying these API endpoints in a configuration >>>> file. Though overriding the API endpoints in the UI's local configuration >>>> file is an option that's available, I understand that we want to move >>>> towards relying exclusively on Keystone for accurate and correct endpoint >>>> configuration. >>>> >>>> Right now, all of the service API endpoints that UI needs to talk with >>>> are only listening on the ctlplane network. >>>> >>>> We've had several iterations of testing and development of the UI over >>>> time and as a result of that, three different solutions that work - >>>> depending on the exact circumstances - have been created which all can >>>> achieve the same goal of allowing the UI to talk to these endpoints: >>>> >>>> - Local SSH port tunneling of the required ports that UI talks to, from >>>> the system running the UI to the Undercloud, and configuring the UI to talk >>>> to localhost:NNNN. This method "works", but it's not a solution we can >>>> recommend >>>> - Making the interface on which these services already listen on - the >>>> ctlplane network - routable. Again, this method "works", but we treat this >>>> interface in a very special manner on purpose, not the least of which >>>> because of it's ability to facilitate pxebooting >>>> - Change the public endpoints in the Keystone catalog to be that of the >>>> existing external, routable interface of the Undercloud for each service >>>> required by the UI. This also requires configuring each service that UI >>>> needs to talk with, to listen on the existing, external, routable interface >>>> on the Undercloud. Some services support a list of interfaces and IPs to >>>> listen on; others require exactly one argument, in which case the address >>>> of 0.0.0.0 would need to be used >>>> >>>> According to the API Endpoint Configuration Recommendation guide[1], >>>> the third option seems most viable and one that we can recommend. The >>>> document briefly describes the security implications of having these >>>> services open on a public interface but recommends the use of a stringent >>>> network policy - something we're used to recommending and helping with. >>>> The first two options, not so much. >>>> >>>> Based on discussions I've had with other people, it's my impression >>>> that the third option is likely the one that we should proceed with. >>>> >>>> This concern is largely transparent to how we're currently testing and >>>> developing the UI because most of that work is done on local, virtualized >>>> environments. When this happens, libvirt does the heavy lifting of >>>> creating a network that's easily routable from the host system. If not >>>> that, then the evolution of instructions for setting up these local >>>> environments over time have recommended using SSH port forwarding. >>>> However, neither of these options should be recommended. >>>> >>>> Thoughts? >>>> >>>> Thanks >>>> -dant >>>> >>>> -- >>>> >>>> P.S. and full disclosure: I'm biased towards the third option. >>>> >>>> >>>> ____________________________________________________________ >>>> ______________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: openstack-dev-requ...@lists.op >>>> enstack.org?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>> >>> >>> -- >>> Juan Antonio Osorio R. >>> e-mail: jaosor...@gmail.com >>> >>> >>> ____________________________________________________________ >>> ______________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.op >>> enstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev