[openstack-dev] [services][tripleo-validations] Process counts for Undercloud services
Hi - tripleo-validations has a validation[0] that attempts to identify services that have more than (in its default setting) 8 processes running when the validation is executed. We've seen in the past, times where too many processes for each service may have had a significant performance impact particularly in Undercloud systems with a less than optimal amount of memory, but it's been a while since I've seen much discussion on this. By default, the validation will look at the following services, and fail if there are more than 8 processes of each running on the Undercloud: - heat-engine - mistral - ironic-inspector - ironic-conductor - nova-api - nova-scheduler - nova-conductor - nova-compute - glance-api - swift-proxy-server - swift-object-server - swift-container-server - zaqar-server Examples of services that this validation fails on immediately out of the box are: - heat-engine - mistral - nova-api What I'm trying to determine right now is if that (default) maximum number of processes (8) is still applicable, and/or reasonable. If not, what max_process_count value would be appropriate? I understand that this question is highly subjective based on a number of factors that go in to the consideration of an OpenStack environment. However, we're looking for some baselines for these values in an effort to make this validation more valuable. Thanks! -dant --- [0] https://github.com/openstack/tripleo-validations/blob/master/validations/undercloud-process-count.yaml __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Need advice on https://review.openstack.org/#/c/521632/
Hi - Yes, planning to keep the manifest, just removing the ping test and the > code that is dependent on that succeeding. > Would the ping test and code be suitable for a TripleO Validation? https://github.com/openstack/tripleo-validations Thanks -dant > > Uploading a new PS to reflect this approach. > > Thanks, > Sandhya > > On 12/11/17, 12:22 PM, "Emilien Macchi" wrote: > > Hi Sandhya, > > See inline: > > On Mon, Dec 11, 2017 at 8:49 AM, Sandhya Dasu (sadasu) < > sad...@cisco.com> wrote: > > Hi Steven and Emilien, > > > > I need your advice on how to proceed with the fix in > > https://review.openstack.org/#/c/521632/. > > > > > > > > The issue in question is that code in puppet-neutron for the Nexus > switch, > > performs a ping test to see if all the Nexus switches specified in > the > > configuration are actually reachable. > > > > After that it performs a ssh-keyscan and adds the list of Nexus > switches to > > the list on known hosts on the Controllers. > > > > Code can be viewed here: > > https://github.com/openstack/puppet-neutron/blob/master/ > manifests/plugins/ml2/cisco/nexus_creds.pp > > (starting from line #104) > > > > I spoke to Emilien about this during the Sydney summit. > > > > > > > > Since then I have tried a bunch of different ways to solve this > problem and > > I am trying to figure out the best way to proceed: > > > > > > > > 1. Adding retry login around the ping test: > > https://review.openstack.org/#/c/521632/2 > > > > 2. Changing the order in which Neutron ML2 plugins/services > were > > initialized in https://review.openstack.org/#/c/521632/8 (Failed > gate > > checks) > > > > 3. I also tried to remove a dependency between the ping test > and the > > ssh-keyscan steps. (code in https://review.openstack.org/# > /c/521632/7) > > > > 4. Finally, in the latest version of the fix I completely > removed the > > ping test and ssh-keyscan steps to make progress. > > (https://review.openstack.org/#/c/521632/) > > > > > > > > Although, the ping test and ssh-keyscan are not essential for the > > functioning of the Nexus driver, I would like to find a way to keep > this > > code. > > > > > > > > Please let me know what would be the best way to proceed. > > IMHO this code shouldn't exist. I've never seen any Puppet code > testing "ping" during a deployment. > You should rather make sure that Nexus switches are ready and > available before performing any deployment with puppet-neutron (with > Ansible for example). > But doing it with Puppet is kind of the wrong way I think. > Let me know if that makes sense but I would rather keep his manifest > to manage config files and remove this code in the future. > -- > Emilien Macchi > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][mistral][deployment] how to add deployment roles
On Mon, Jul 3, 2017 at 7:08 AM, Alex Schultz wrote: > On Tue, Jun 27, 2017 at 2:19 PM, Dan Trainor wrote: > > Hi - > > > > I'm looking for the glue that populates the overcloud role list. > > > > I can get a list of roles via 'openstack overcloud role list', however > I'm > > looking to create new roles to incorporate in to this list. > > > > I got as far as using 'mistral action-update' against what I beleive to > be > > the proper action (tripleo.role.list) but am not sure what to use as the > > source of what I would be updating, not am I finding any information > about > > how that runs and where it gets its data from. I also had a nice > exercise > > pruning the output of 'mistral action-*' commands which was pretty > > insightful and helped me hone in on what I was looking for, but still > > uncertain of. > > > > Pretty sure I'm missing a few details along the way here, too. > > > > Can someone please shed some light on this so I can have a better > > understanding of the process? > > > > Sorry for the delay in replying (pto). The overcloud roles actions > operate on a folder. By default this is > /usr/share/openstack-tripleo-heat-templates/roles. However it does > provide a command line parameter to change this folder to where ever > you would like to keep your custom roles. The docs around this are > currently up for review[0] but we do have an existing readme in the > roles folder[1] that has some of this information. The plan is to > eventually allow for some mistral actions around roles, but we > probably won't have that until later. Right now this is purely a > tripleoclient function. > > Thanks, > -Alex > > [0] https://review.openstack.org/#/c/476236/ > [1] https://github.com/openstack/tripleo-heat-templates/blob/ > master/roles/README.rst > > Alex - Thanks so much for the reply. The documentation is spot-on and provides a great introduction to role manipulation for Overcloud deployment. Now that I have a better context for what I'm trying to ask, I think I want to clarify that my request is more in line with the role types that are available for assignment prior to deployment. In the TripleO UI, several Roles are available for assigning nodes: Block Storage, Ceph Storage, Compute, Controller, and Object Storage. My goal is to create an additional role for assignment in this dialogue. I understand now, based on your feedback and that of Steve Hardy, I can manipulate roles per the Overcloud deployment. However, I'm interested in manipulating the list of roles that are available for assignment per the UI. I was hoping the UI was aligned close enough with tripleoclient to provide a straight-forward interface for this, but I may be wrong. This might be one for the UI guys. I'll start pinging them in their timezone. But in the mean time if you have any suggestions, I would appreciate the feedback. Thanks! -dant __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][mistral][deployment] how to add deployment roles
HI, Steve - > I think perhaps the confusion is because this was implemented in > tripleoclient, and porting it to tripleo-common is not yet completed? > (Alex can confirm the status of this but it was planned I think). > > Related ML discussion which includes links to the patches: > > http://lists.openstack.org/pipermail/openstack-dev/2017-June/118157.html > > http://lists.openstack.org/pipermail/openstack-dev/2017-June/118213.html > > Thanks for digging this up. I do vaguely remember seeing this. This points me more in the direction I'm looking for. Thanks! -dant __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [heat][mistral][deployment] how to add deployment roles
Hi - I'm looking for the glue that populates the overcloud role list. I can get a list of roles via 'openstack overcloud role list', however I'm looking to create new roles to incorporate in to this list. I got as far as using 'mistral action-update' against what I beleive to be the proper action (tripleo.role.list) but am not sure what to use as the source of what I would be updating, not am I finding any information about how that runs and where it gets its data from. I also had a nice exercise pruning the output of 'mistral action-*' commands which was pretty insightful and helped me hone in on what I was looking for, but still uncertain of. Pretty sure I'm missing a few details along the way here, too. Can someone please shed some light on this so I can have a better understanding of the process? Thanks! -dant __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [validations][ui] resetting state of validations
Hi - Is there a way to reset the state of all the validations that have previously ran, back to the original state they were prior to running? Using the UI, for example, some validations (by design) run as soon as you log in. Others run after different actions are completed. But there's a state at which none of the validations have been ran, prior to logging in to the UI. I want to re-run those validations as if I had logged in to the UI for the first time, for testing purposes. Thanks! -dant __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Attempting to proxy websockets through Apache or HAProxy for Zaqar
Hi - In an attempt to work on [0], I've been playing around with proxying all the service API endpoints that the UI needs to communicate with, through either haproxy or Apache to avoid a bug[1] around how non-Chrome browsers handle SSL connections to different ports on the same domain. The blueprint suggests using haproxy for this, but we're currently using the "old" notation of listen/server, not frontend/backend. The distinction is important because the ACLs that would allow any kind of proxying to facilitate this are only available in the latter notation. In order to do this in haproxy, tripleo::haproxy would need a rewrite (looks pretty trivial, but likely out of scope for this). So I'd really like to isolate this to UI, which is convenient since UI runs largely self-contained inside Apache. I've made some good progress with most all of the services, since they were pretty straight-forward - mod_proxy handles them just fine. The one I'm not able to make work right now is the websocket service that UI uses. Ultimately, I see the Websocket connection get upgraded and the Websocket opens, but stays open indefinitely and will never see more than 0 bytes. No data is transferred from the browser over the Websocket. This connection hangs indefinitely, and UI does not complete any operations that depend on the Zaqar Websocket. Observing trace6[4] output, I can see mod_proxy_wstunnel (which relies on mod_proxy) make the connection, I can see Zaqar recognize the request in logs, the client (UI) doesn't send or receive any data from it. It's as if immediately after the Upgrade[2], the persistent Websocket connection just dies. I've had limited success using a couple different implementations of this in Apache. ProxyPass/ProxyPassReverse looks as if it should work (so long as mod_proxy_wstunnel is loaded), but this is not my experience. Using a mod_rewrite rule[3] to force the specific Websocket proxy for a specific URI (/zaqar) has the same outcome. In its most simple form, the ProxyPass rule I'm attempting to use is: ProxyPass "/zaqar""ws://192.0.2.1:9000/" ProxyPassReverse "/zaqar""ws://192.0.2.1:9000/" Note that I've used several variations of both ProxyPass configurations and mod_rewrite rules using the [P] flag which all seem to net the same result. I've also tried writing the same functional equivalent in haproxy using a frontend/backend notation to confirm if this was a protocol thing or a software thing (if haproxy could do this, but Apache could not). >From the top, here's some Apache logs (note that trace6 is noisy, I just grep'd for ws, wss, and the websocket port (9000); full logs of this request are [4]): [Tue Jan 17 12:08:16.639170 2017] [proxy_wstunnel:debug] [pid 32128] mod_proxy_wstunnel.c(253): [client 192.0.2.1:51508] AH02445: woke from poll(), i=1 [Tue Jan 17 12:08:16.639220 2017] [proxy_wstunnel:debug] [pid 32128] mod_proxy_wstunnel.c(278): [client 192.0.2.1:51508] AH02448: client was readable [Tue Jan 17 12:08:16.639265 2017] [core:trace6] [pid 32128] core_filters.c(525): [remote 192.0.2.1:9000] core_output_filter: flushing because of FLUSH bucket [Tue Jan 17 12:08:16.639337 2017] [proxy_wstunnel:trace2] [pid 32128] mod_proxy_wstunnel.c(295): [client 192.0.2.1:51508] finished with poll() - cleaning up [Tue Jan 17 12:08:16.640023 2017] [proxy:debug] [pid 32128] proxy_util.c(2218): AH00943: WS: has released connection for (192.0.2.1) [Tue Jan 17 12:08:19.238044 2017] [core:trace5] [pid 32128] protocol.c(618): [client 192.0.2.1:51996] Request received from client: GET /zaqar HTTP/1.1 [Tue Jan 17 12:08:19.238191 2017] [core:trace3] [pid 32128] request.c(293): [client 192.0.2.1:51996] request authorized without authentication by access_checker_ex hook: /zaqar [Tue Jan 17 12:08:19.238202 2017] [proxy_wstunnel:trace1] [pid 32128] mod_proxy_wstunnel.c(51): [client 192.0.2.1:51996] canonicalising URL // 192.0.2.1:9000/ [Tue Jan 17 12:08:19.238223 2017] [proxy:trace2] [pid 32128] proxy_util.c(1985): [client 192.0.2.1:51996] ws: found worker ws:// 192.0.2.1:9000/ for ws://192.0.2.1:9000/ [Tue Jan 17 12:08:19.238227 2017] [proxy:debug] [pid 32128] mod_proxy.c(1117): [client 192.0.2.1:51996] AH01143: Running scheme ws handler (attempt 0) [Tue Jan 17 12:08:19.238231 2017] [proxy_http:debug] [pid 32128] mod_proxy_http.c(1925): [client 192.0.2.1:51996] AH01113: HTTP: declining URL ws://192.0.2.1:9000/ [Tue Jan 17 12:08:19.238236 2017] [proxy_wstunnel:debug] [pid 32128] mod_proxy_wstunnel.c(333): [client 192.0.2.1:51996] AH02451: serving URL ws://192.0.2.1:9000/ [Tue Jan 17 12:08:19.238239 2017] [proxy:debug] [pid 32128] proxy_util.c(2203): AH00942: WS: has acquired connection for (192.0.2.1) [Tue Jan 17 12:08:19.238244 2017] [proxy:debug] [pid 32128] proxy_util.c(2256): [client 192.0.2.1:51996] AH00944: connecting ws:// 192.0.2.1:9000/ to 192.0.2.1:9000 [Tue Jan 17 12:08:19.238249 2017] [proxy:debug] [pid 32128] proxy_util.c(2422): [client 192.0.2.1:51
Re: [openstack-dev] [tripleo] Proposing Alex Schultz core on puppet-tripleo
+1 On Dec 1, 2016 5:36 PM, "James Slagle" wrote: > On Thu, Dec 1, 2016 at 5:26 PM, Emilien Macchi wrote: > > Team, > > > > Alex Schultz (mwhahaha on IRC) has been active on TripleO since a few > > months now. While he's very active in different areas of TripleO, his > > reviews and contributions on puppet-tripleo have been very useful. > > Alex is a Puppet guy and also the current PTL of Puppet OpenStack. I > > think he perfectly understands how puppet-tripleo works. His > > involvement in the project and contributions on puppet-tripleo deserve > > that we allow him to +2 puppet-tripleo. > > > > Thanks Alex for your involvement and hard work in the project, this is > > very appreciated! > > > > As usual, I'll let the team to vote about this proposal. > > +1 > > > -- > -- James Slagle > -- > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Proposing Julie Pichon for tripleo core
On Tue, Nov 22, 2016 at 11:36 AM, Michele Baldessari wrote: > On Tue, Nov 22, 2016 at 05:01:49PM +, Dougal Matthews wrote: > > Hi all, > > > > I would like to propose we add Julie (jpich) to the TripleO core team for > > python-tripleoclient and tripleo-common. This nomination is based > partially > > on review stats[1] and also my experience with her reviews and > > contributions. > > > > Julie has consistently provided thoughtful and detailed reviews since the > > start of the Newton cycle. She has made a number of contributions which > > improve the CLI and has been extremely helpful with other tasks that > don't > > often get enough attention (backports, bug triaging/reporting and > improving > > our processes[2]). > > > > I think she will be a valuable addition to the review team > +1 couldn't agree more __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Getting the UI to talk with Undercloud API service endpoints
Hi - Great suggestions, Dan. To recap, we followed that up with a few other ideas on irc and we eventually came to a point to test some of this, with slight modification. UI also ships with a configuration file that can override the endpoint information received from Keystone. The file is located at /var/www/openstack-tripleo-ui/dist/tripleo_ui_config.js. Part of making this work means enabling SSL on the Undercloud when the UI component is selected for installation in undercloud.conf. I think this is going to be a pretty reasonable request, but I'm interested in hearing feedback from this, and what other implications it may have, that I can't think of. The changes I made were all one-off, unmanaged changes, just to test this idea out. I'll be doing some more tests but will probably be looking for acceptance shortly. Once SSL was enabled on the Undercloud, I made two edits to haproxy.cfg that were pretty straightforward: added a 'listen' server directive for UI to both terminate SSL and forward the request to Apache, and added a 'bind' statement for each service that UI expects to talk to (keystone, heat, ironic, mistral, swift, zaqar-websocket). Once those configuration changes were made, I had a very pleasant experience using the UI. It worked exactly as expected. I think this might be a viable option. Thoughts? Thanks! -dant On Fri, Sep 30, 2016 at 12:21 PM, Dan Sneddon wrote: > Thinking about this a little more, creating a new unified endpoint on the > same port as the UI doesn't solve the problem at hand. The UI will use the > service catalog to find endpoints, so we would need to change the endpoints > in the service catalog, which changes the flow for the underlying services > as well. > > Simply moving the control plane services to the external network won't > work well in environments where the control plane is isolated and > non-routed. The Undercloud can forward packets, but then becomes a > bottleneck and a SPOF. > > A few approaches come to mind, but none of these are quick fixes: > > * Change the UI to get its list of endpoints from somewhere other than the > service catalog and customize this with URLs that point to the Public VIP. > Duplicate the services required for the UI on both control plane and > external network. This might make it possible to make all connections over > port 443, which is more firewall-friendly (which would be desirable or not > depending on what kind of firewalling and traffic management is wanted). > > * Relax the rp_filter settings on the Controllers so they accept packets > destined for the external network on their control plane interfaces; add a > static route to the Public VIP via the control plane VIP on all > non-controller nodes. Modify the service catalog to point to the public VIP > for the services the UI needs. This would need to be combined with a > security review to determine if additional iptables rules are required. > > * Split the service catalog, so we have an internal and an external view > depending on where the query came from. I'm not sure how feasible this is. > > Of these, I think the rp_filter settings are the only ones that could be > done solely with TripleO code changes. That might be worth investigating. > > Dan Sneddon | Principal OpenStack Engineer | dsned...@redhat.com > > > On Sep 30, 2016, at 11:36 AM, Dan Sneddon wrote: > > I don't think we can rely on the Undercloud as an API proxy unless we > address the lack of HA on the Undercloud. > > Wouldn't this be better implemented as as a single, name-based HAProxy > instance running SSL on port 443 on the overcloud Public VIP? Then we could > have the same endpoint for Horizon and every other API. > > I actually implemented this scheme in Havana before I joined Red Hat. At > the time, we had to have a complex HAProxy config and patch the end points > to support name-based URLs. I think some work has been done in OpenStack > now to support this model, but I'm not sure where it stands. > > Dan Sneddon | Principal OpenStack Engineer | dsned...@redhat.com > > > On Sep 30, 2016, at 9:44 AM, Dan Trainor wrote: > > Hi - > > I re-read your email a few times and like a few of the things that I see, > but I'd love some more clarification. As I read it, a few of these things > conflict. I believe you're suggesting that we don't make these services > listen on a public interface due to security concerns (and of course, > enabling SSL would break this because haproxy would listen on these > interfaces/ports), but this approach would be acceptable if HAProxy was > listening on these ports, terminating SSL, and sending them to each > respective service backend. Am I correct i underst
Re: [openstack-dev] [tripleo] Getting the UI to talk with Undercloud API service endpoints
Hi - I re-read your email a few times and like a few of the things that I see, but I'd love some more clarification. As I read it, a few of these things conflict. I believe you're suggesting that we don't make these services listen on a public interface due to security concerns (and of course, enabling SSL would break this because haproxy would listen on these interfaces/ports), but this approach would be acceptable if HAProxy was listening on these ports, terminating SSL, and sending them to each respective service backend. Am I correct i understanding this? Are you suggesting that these endpoint ports would still be externally accessible on the primary (public) interface of the Undercloud, but just managed by HAProxy? I think that's an acceptable approach. Even if these endpoints are, like you suggested, listening on a separate network or IP as the Undercloud's primary interface, at least then it would be easier for organizations to enforce network access policies to these ports, and subsequently, these services that UI needs to talk to directly. I'm also perfectly fine with suggesting that if UI is installed, then this forces the Undercloud to be SSL enabled. This would be a good way to move the idea of a secured, by default SSL-enabled Undercloud forward a little more, which is something we'd definitely like to see more. Thoughts? Thanks -dant On Thu, Sep 29, 2016 at 9:01 AM, Dan Trainor wrote: > Hi, Juan - > > Actually, the third option is also not an option in the current undercloud >> setup, since making the services listen in 0.0.0.0 will break HAProxy. So >> when you're deploying with TLS things will break since we use HAProxy to >> terminate TLS connections. >> > > Ah, that's correct, isn't it. > > > >> On the other hand, we also don't want services to listen on 0.0.0.0 since >> that would become a security concern. We should instead be blocking >> everything we don't need to have exposed (as we've done with the >> undercloud's database and rabbitmq). >> > > I don't disagree that we want to focus on least privilege, but we do have > documentation that talks about how to deal with this. I addressed this in > my previous message, even if only to illustrate my understanding that there > would be concerns around this. See more comments about this down below... > > >> >> Now, as we're trying to move to have more convergence between the >> undercloud and the overcloud (trying to deploy the undercloud via heat >> also, as Dan Prince has mentioned), I think some aspecs of this will bring >> a solution to this problem. for instance, just like we already do in the >> overcloud, we could have the undercloud's HAProxy always terminate the >> endpoints, which I'm attempting with these two patches: >> https://review.openstack.org/#/c/360366 https://review.openstack.org/# >> /c/360368 . Furthermore, we could have the public endpoints in HAProxy >> listen on a separate network that's accessible externally, also as we do in >> the overcloud. That way we don't need to change the actual interfaces the >> services are listening on, and rely on HAProxy, getting closer to how we do >> things in the overcloud. It seems to me that it would help solve the >> problem. >> > > I like that idea. Though, this effectively shifts the process of having > these services themselves listen on different IPs/ports and offloads that > to HAProxy. Whatever security concerns we have with opening up network > communications would still exist, but I think that would be more broadly > accepted considering these connections are now managed by HAProxy which > *only* listens for SSL connections. > > Another challenge with isolating this traffic on a separate network is > that we introduce a new dependency that's going to take administrative time > to set up and configure outside of OpenStack - do we really want that? > > Thanks! > -dant > > > > >> >> On Wed, Sep 28, 2016 at 7:24 PM, Dan Trainor wrote: >> >>> Hi - >>> >>> I want to bring up a subject that needs a little more attention. There >>> are a few ideas floating around but it's important that we get this done >>> right. >>> >>> UI is unique in the sense that it operates almost entirely in a browser, >>> talking directly to service API endpoints which it either figures out from >>> they Keystone service catalog as using the publicURL endpoint for each >>> service, or by specifying these API endpoints in a configuration file. >>> Though overriding the API endpoints in the UI's local configuration
Re: [openstack-dev] [tripleo] Getting the UI to talk with Undercloud API service endpoints
Hi, Juan - Actually, the third option is also not an option in the current undercloud > setup, since making the services listen in 0.0.0.0 will break HAProxy. So > when you're deploying with TLS things will break since we use HAProxy to > terminate TLS connections. > Ah, that's correct, isn't it. > On the other hand, we also don't want services to listen on 0.0.0.0 since > that would become a security concern. We should instead be blocking > everything we don't need to have exposed (as we've done with the > undercloud's database and rabbitmq). > I don't disagree that we want to focus on least privilege, but we do have documentation that talks about how to deal with this. I addressed this in my previous message, even if only to illustrate my understanding that there would be concerns around this. See more comments about this down below... > > Now, as we're trying to move to have more convergence between the > undercloud and the overcloud (trying to deploy the undercloud via heat > also, as Dan Prince has mentioned), I think some aspecs of this will bring > a solution to this problem. for instance, just like we already do in the > overcloud, we could have the undercloud's HAProxy always terminate the > endpoints, which I'm attempting with these two patches: > https://review.openstack.org/#/c/360366 https://review.openstack.org/# > /c/360368 . Furthermore, we could have the public endpoints in HAProxy > listen on a separate network that's accessible externally, also as we do in > the overcloud. That way we don't need to change the actual interfaces the > services are listening on, and rely on HAProxy, getting closer to how we do > things in the overcloud. It seems to me that it would help solve the > problem. > I like that idea. Though, this effectively shifts the process of having these services themselves listen on different IPs/ports and offloads that to HAProxy. Whatever security concerns we have with opening up network communications would still exist, but I think that would be more broadly accepted considering these connections are now managed by HAProxy which *only* listens for SSL connections. Another challenge with isolating this traffic on a separate network is that we introduce a new dependency that's going to take administrative time to set up and configure outside of OpenStack - do we really want that? Thanks! -dant > > On Wed, Sep 28, 2016 at 7:24 PM, Dan Trainor wrote: > >> Hi - >> >> I want to bring up a subject that needs a little more attention. There >> are a few ideas floating around but it's important that we get this done >> right. >> >> UI is unique in the sense that it operates almost entirely in a browser, >> talking directly to service API endpoints which it either figures out from >> they Keystone service catalog as using the publicURL endpoint for each >> service, or by specifying these API endpoints in a configuration file. >> Though overriding the API endpoints in the UI's local configuration file is >> an option that's available, I understand that we want to move towards >> relying exclusively on Keystone for accurate and correct endpoint >> configuration. >> >> Right now, all of the service API endpoints that UI needs to talk with >> are only listening on the ctlplane network. >> >> We've had several iterations of testing and development of the UI over >> time and as a result of that, three different solutions that work - >> depending on the exact circumstances - have been created which all can >> achieve the same goal of allowing the UI to talk to these endpoints: >> >> - Local SSH port tunneling of the required ports that UI talks to, from >> the system running the UI to the Undercloud, and configuring the UI to talk >> to localhost:. This method "works", but it's not a solution we can >> recommend >> - Making the interface on which these services already listen on - the >> ctlplane network - routable. Again, this method "works", but we treat this >> interface in a very special manner on purpose, not the least of which >> because of it's ability to facilitate pxebooting >> - Change the public endpoints in the Keystone catalog to be that of the >> existing external, routable interface of the Undercloud for each service >> required by the UI. This also requires configuring each service that UI >> needs to talk with, to listen on the existing, external, routable interface >> on the Undercloud. Some services support a list of interfaces and IPs to >> listen on; others require exactly one argument, in which case the address >> of
[openstack-dev] [tripleo] Getting the UI to talk with Undercloud API service endpoints
Hi - I want to bring up a subject that needs a little more attention. There are a few ideas floating around but it's important that we get this done right. UI is unique in the sense that it operates almost entirely in a browser, talking directly to service API endpoints which it either figures out from they Keystone service catalog as using the publicURL endpoint for each service, or by specifying these API endpoints in a configuration file. Though overriding the API endpoints in the UI's local configuration file is an option that's available, I understand that we want to move towards relying exclusively on Keystone for accurate and correct endpoint configuration. Right now, all of the service API endpoints that UI needs to talk with are only listening on the ctlplane network. We've had several iterations of testing and development of the UI over time and as a result of that, three different solutions that work - depending on the exact circumstances - have been created which all can achieve the same goal of allowing the UI to talk to these endpoints: - Local SSH port tunneling of the required ports that UI talks to, from the system running the UI to the Undercloud, and configuring the UI to talk to localhost:. This method "works", but it's not a solution we can recommend - Making the interface on which these services already listen on - the ctlplane network - routable. Again, this method "works", but we treat this interface in a very special manner on purpose, not the least of which because of it's ability to facilitate pxebooting - Change the public endpoints in the Keystone catalog to be that of the existing external, routable interface of the Undercloud for each service required by the UI. This also requires configuring each service that UI needs to talk with, to listen on the existing, external, routable interface on the Undercloud. Some services support a list of interfaces and IPs to listen on; others require exactly one argument, in which case the address of 0.0.0.0 would need to be used According to the API Endpoint Configuration Recommendation guide[1], the third option seems most viable and one that we can recommend. The document briefly describes the security implications of having these services open on a public interface but recommends the use of a stringent network policy - something we're used to recommending and helping with. The first two options, not so much. Based on discussions I've had with other people, it's my impression that the third option is likely the one that we should proceed with. This concern is largely transparent to how we're currently testing and developing the UI because most of that work is done on local, virtualized environments. When this happens, libvirt does the heavy lifting of creating a network that's easily routable from the host system. If not that, then the evolution of instructions for setting up these local environments over time have recommended using SSH port forwarding. However, neither of these options should be recommended. Thoughts? Thanks -dant -- P.S. and full disclosure: I'm biased towards the third option. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
