[openstack-dev] [nova] vendordata plugin for freeIPA host enrollment
Wanted to let you know I'm working on a nova metadata vendordata plugin that will help automate instance enrollment into a freeIPA server. This will do a number of things for a user: - provide centralized user identity, sudo and host-based access control for the instances - provide the instance an identity it can use for itself - using this identity a host can obtain SSL certificates for itself from your freeIPA CA If ipa_enroll is set to True in the instance metadata (or in the image metadata) when a nova instance is spawned then a one-time password will be created and IPA enrollment will occur during the cloud-init stage. Code is currently at https://github.com/rcritten/novajoin rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][keystone] auth for new metadata plugins
Adam Young wrote: On 08/15/2016 05:10 PM, Rob Crittenden wrote: Review https://review.openstack.org/#/c/317739/ added a new dynamic metadata handler to nova. The basic jist is that rather than serving metadata statically, it can be done dyamically, so that certain values aren't provided until they are needed, mostly for security purposes (like credentials to enroll in an AD domain). The metadata is configured as URLs to a REST service. Very little is passed into the REST call, mostly UUIDs of the instance, image, etc. to ensure a stable API. What this means though is that the REST service may need to make calls into nova or glance to get information, like looking up the image metadata in glance. Currently the dynamic metadata handler _can_ generate auth headers if an authenticated request is made to it, but consider that a common use case is fetching metadata from within an instance using something like: % curl http://169.254.169.254/openstack/2016-10-06/vendor_data2.json This will come into the nova metadata service unauthenticated. So a few questions: 1. Is it possible to configure paste (I'm a relative newbie) both authenticated and unauthenticated requests are accepted such that IF an authenticated request comes it, those credentials can be used, otherwise fall back to something else? Only if they are on different URLs, I think. Its auth_token middleware for all services but Keystone. Keystone, the rles are similar, but the implementation is a little different. Ok. I'm fine with the unauthenticated path if the service we can just create a separate service user for it. 2. If an unauthenticated request comes in, how best to obtain a token to use? Is it best to create a service user for the REST services (perhaps several), use a shared user, something else? No unauthenticated requests, please. If the call is to Keystone, we could use the X509 Tokenless approach, but if the call comes from the new server, you won't have a cert by the time you need to make the call, will you? Not sure which cert you're referring too but yeah, the metadata service is unauthenticated. The requests can come in from the instance which has no credentials (via http://169.254.169.254/). Shared service users are probably your best bet. We can limit the roles that they get. What are these calls you need to make? To glance for image metadata, Keystone for project information and nova for instance information. The REST call passes in various UUIDs for these so they need to be dereferenced. There is no guarantee that these would be called in all cases but it is a possibility. rob I guess if config_drive is True then this isn't really a problem as the metadata will be there in the instance already. thanks rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova][keystone] auth for new metadata plugins
Review https://review.openstack.org/#/c/317739/ added a new dynamic metadata handler to nova. The basic jist is that rather than serving metadata statically, it can be done dyamically, so that certain values aren't provided until they are needed, mostly for security purposes (like credentials to enroll in an AD domain). The metadata is configured as URLs to a REST service. Very little is passed into the REST call, mostly UUIDs of the instance, image, etc. to ensure a stable API. What this means though is that the REST service may need to make calls into nova or glance to get information, like looking up the image metadata in glance. Currently the dynamic metadata handler _can_ generate auth headers if an authenticated request is made to it, but consider that a common use case is fetching metadata from within an instance using something like: % curl http://169.254.169.254/openstack/2016-10-06/vendor_data2.json This will come into the nova metadata service unauthenticated. So a few questions: 1. Is it possible to configure paste (I'm a relative newbie) both authenticated and unauthenticated requests are accepted such that IF an authenticated request comes it, those credentials can be used, otherwise fall back to something else? 2. If an unauthenticated request comes in, how best to obtain a token to use? Is it best to create a service user for the REST services (perhaps several), use a shared user, something else? I guess if config_drive is True then this isn't really a problem as the metadata will be there in the instance already. thanks rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] How to enable SSL in devStack?
Clark Boylan wrote: On Wed, Jul 20, 2016, at 07:01 AM, Rob Crittenden wrote: Andrey Pavlov wrote: Hi, When I ran devstack with SSL I found a bug and tried to fix it - https://review.openstack.org/#/c/242812/ But no one agree with me. Try to apply this patch - it may help. Also there is a chance that new bugs present in devstack that prevented to install it with SSL. Seeing how some other things in your local.conf might help but when I tried to reproduce it I got the same error and it failed because Apache didn't have an SSL listener on 443. I'm not sure I'd recommend direct SSL in any case. I'd recommend the tls-proxy service instead. Note that I'm pretty sure it has the same problem: it hasn't been updated to handle port 443 for Keystone. I pushed a change up (https://review.openstack.org/#/c/296771/) to enable tls-proxy in devstack-gate to see how it does and it wasn't too happy. Is it worth trying to make a push on this and just enabling it by default in devstack? The failure is due to the Keystone switch to using URLs in favor of ports to distinguish user and admin operations. The fix is fairly straightforward and I have it fixed in a related change, switching from stud to mod_proxy https://review.openstack.org/#/c/301172 I'd be fine making the tls-proxy gate job voting once we get things working again. rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] How to enable SSL in devStack?
Brant Knudson wrote: On Wed, Jul 20, 2016 at 12:29 PM, Rob Crittenden <rcrit...@redhat.com <mailto:rcrit...@redhat.com>> wrote: Fixing Keystone is easy. An Apache VirtualHost for 443 needs to be added. But I found another, deeper problem: cinder won't listen on SSL. When they switched to using oslo_service for WSGI they completely removed the ability to use SSL. See bug https://bugs.launchpad.net/cinder/+bug/1590901 rob Problems like this should make us wonder why we're reimplementing basic functionality like TLS termination. Existing wsgi containers (uwsgi, gunicorn, and apache) all handle TLS termination just fine. I'm not exactly sure what you mean. If you mean that doing native TLS in eventlet is not a great idea then we are in agreement. But to remove it will should require a plan, not an unexpected side-effect of another change. rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] How to enable SSL in devStack?
Rob Crittenden wrote: Andrey Pavlov wrote: Hi, When I ran devstack with SSL I found a bug and tried to fix it - https://review.openstack.org/#/c/242812/ But no one agree with me. Try to apply this patch - it may help. Also there is a chance that new bugs present in devstack that prevented to install it with SSL. Seeing how some other things in your local.conf might help but when I tried to reproduce it I got the same error and it failed because Apache didn't have an SSL listener on 443. I'm not sure I'd recommend direct SSL in any case. I'd recommend the tls-proxy service instead. Note that I'm pretty sure it has the same problem: it hasn't been updated to handle port 443 for Keystone. I'm working on switching from stud to mod_proxy if you want to take a look and this problem is fixed there, https://review.openstack.org/301172 I'll see about adding a SSL listener to Keystone for the USE_SSL case in the next few days. And yeah, it's a moving target. I have an experimental gate test for tlsproxy but it has to be requested explicitly. My plan is to enable it as non-voting once the mod_proxy changes land so it will at least be more obvious when things break (or maybe we can making it voting). Fixing Keystone is easy. An Apache VirtualHost for 443 needs to be added. But I found another, deeper problem: cinder won't listen on SSL. When they switched to using oslo_service for WSGI they completely removed the ability to use SSL. See bug https://bugs.launchpad.net/cinder/+bug/1590901 rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] How to enable SSL in devStack?
Andrey Pavlov wrote: Hi, When I ran devstack with SSL I found a bug and tried to fix it - https://review.openstack.org/#/c/242812/ But no one agree with me. Try to apply this patch - it may help. Also there is a chance that new bugs present in devstack that prevented to install it with SSL. Seeing how some other things in your local.conf might help but when I tried to reproduce it I got the same error and it failed because Apache didn't have an SSL listener on 443. I'm not sure I'd recommend direct SSL in any case. I'd recommend the tls-proxy service instead. Note that I'm pretty sure it has the same problem: it hasn't been updated to handle port 443 for Keystone. I'm working on switching from stud to mod_proxy if you want to take a look and this problem is fixed there, https://review.openstack.org/301172 I'll see about adding a SSL listener to Keystone for the USE_SSL case in the next few days. And yeah, it's a moving target. I have an experimental gate test for tlsproxy but it has to be requested explicitly. My plan is to enable it as non-voting once the mod_proxy changes land so it will at least be more obvious when things break (or maybe we can making it voting). rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Are injected files in compute going to be deprecated?
In nova/compute/manager.py I see: def inject_file(self, context, path, file_contents, instance): """Write a file to the specified path in an instance on this host.""" # NOTE(russellb) Remove this method, as well as the underlying virt # driver methods, when the compute rpc interface is bumped to 4.x # as it is no longer used. The RPC API is at 4.5 as of Liberty. Does that mean it is going soon? I couldn't find any deprecation references other than this one. rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] nova hooks - document & test or deprecate?
Daniel P. Berrange wrote:
> On Mon, Feb 29, 2016 at 11:59:06AM -0500, Sean Dague wrote:
>> The nova/hooks.py infrastructure has been with us since early Nova. It's
>> currently only annotated on a few locations - 'build_instance',
>> 'create_instance', 'delete_instance', and 'instance_network_info'. It's
>> got a couple of unit tests on it, but nothing that actually tests real
>> behavior of the hooks we have specified.
>>
>> It does get used in the wild, and we do break it with changes we didn't
>> ever anticipate would impact it -
>> https://bugs.launchpad.net/nova/+bug/1518321
>>
>> However, when you look into how that is used, it's really really odd and
>> fragile -
>> https://github.com/richm/rdo-vm-factory/blob/master/rdo-ipa-nova/novahooks.py#L248
>>
>>
>> def pre(self, *args, **kwargs):
>> # args[7] is the injected_files parameter array
>> # the value is ('filename', 'base64 encoded contents')
>> ipaotp = str(uuid.uuid4())
>> ipainject = ('/tmp/ipaotp', base64.b64encode(ipaotp))
>> args[7].extend(self.inject_files)
>> args[7].append(ipainject)
>>
>> In our continued quest on being more explicit about plug points it feels
>> like we should other document the interface (which means creating
>> stability on the hook parameters) or we should deprecate this construct
>> as part of a bygone era.
>>
>> I lean on deprecation because it feels like a thing we don't really want
>> to support going forward, but I can go either way.
>
> As it is designed, I think the hooks mechanism is really unsupportable
> long term. It is exposing users to arbitrary internal Nova data structures
> which we have changed at will and we cannot possibly ever consider them
> to be a stable consumable API. I'm rather surprised we've not seen more
> bugs like the one you've shown above - most likely thats a reflection
> on not many people actually using this facility.
>
> I'd be strongly in favour of deprecation & removal of this hooking
> mechanism, as its unsupportable in any sane manner when it exposes
> code to our internal unstable API & object model.
>
> If there's stuff people are doing in hooks that's impossible any other
> way, we should really be looking at what we need to provide in our
> public API to achieve the same goal, if it is use case we wish to be
> able to support.
I'll just add that lots and lots of software has hooks. Just because the
initial implementation decided to expose internal structures (which lots
of other APIs do in order to be interesting) doesn't make it inherently
bad. It just requires a stable API rather than passing *args, **kwargs
around. It seems like the original design was to throw in the kitchen
sink when a more targeted set of arguments would probably do just fine.
rob
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] nova hooks - document & test or deprecate?
Daniel P. Berrange wrote: > On Mon, Feb 29, 2016 at 12:36:03PM -0700, Rich Megginson wrote: >> On 02/29/2016 12:19 PM, Chris Friesen wrote: >>> On 02/29/2016 12:22 PM, Daniel P. Berrange wrote: >>> There's three core scenarios for hooks 1. Modifying some aspect of the Nova operation 2. Triggering an external action synchronously to some Nova operation 3. Triggering an external action asynchronously to some Nova operation The Rdo example is falling in scenario 1 since it is modifying the injected files. I think this is is absolutely the kind of thing we should explicitly *never* support. When external code can arbitrarily modify some aspect of Nova operation we're in totally unchartered territory as to the behaviour of Nova. To support that we'd need to provide a stable internal API which is just not something we want to tie ourselves into. I don't know just what the Rdo example is trying to achieve, but whatever it is, it should be via some supportable API and not a hook., Scenaris 2 and 3 are both valid to consider. Using the notifications system gets as an asynchronous trigger mechanism, which is probably fine for many scenarios. The big question is whether there's a compelling need for scenario two, where the external action blocks execution of the Nova operation until it has completed its hook. >>> >>> Even in the case of scenario two it is possible in some cases to use a >>> proxy to intercept the HTTP request, take action, and then forward it or >>> reject it as appropriate. >>> >>> I think the real question is whether there's a need to trigger an external >>> action synchronously from down in the guts of the nova code. >> >> The hooks do the following: >> https://github.com/rcritten/rdo-vm-factory/blob/use-centos/rdo-ipa-nova/novahooks.py#L271 >> >> We need to generate a token (ipaotp) and call ipa host-add with that token >> _before_ the new machine has a chance to call ipa-client-install. We have >> to guarantee that the client cannot call ipa-client-install until we get >> back the response from ipa that the host has been added with the token. >> Additionally, we store the token in an injected_file in the new machine, so >> the file can be removed as soon as possible. We tried storing the token in >> the VM metadata, but there is apparently no way to delete it? Can the >> machine do >> >> curl -XDELETE http://168.254.169.254/openstack/latest/metadata?key=value ? >> >> Using the build_instance.pre hook in Nova makes this simple and >> straightforward. It's also relatively painless to use the network_info.post >> hook to handle the floating ip address assignment. >> >> Is it possible to do the above using notifications without jumping through >> too many hoops? > > So from a high level POV, you are trying to generate a security token > which will be provided to the guest OS before it is booted. > > I think that is a pretty clearly useful feature, and something that > should really be officially integrated into Nova as a concept rather > than done behind nova's back as a hook. Note that the reason the file was injected the way it was is so that Nova would have no idea there even is a token. We didn't want someone later peeking at the metadata, or a notification, to get the token. rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] nova hooks - document & test or deprecate?
Andrew Laski wrote: > > > On Mon, Feb 29, 2016, at 12:12 PM, Dan Smith wrote: >>> In our continued quest on being more explicit about plug points it feels >>> like we should other document the interface (which means creating >>> stability on the hook parameters) or we should deprecate this construct >>> as part of a bygone era. >>> >>> I lean on deprecation because it feels like a thing we don't really want >>> to support going forward, but I can go either way. >> >> Deprecate and remove, please. We've been removing these sorts of things >> over time, and nova hooks have been ignored in that process. But really, >> making them more rigid is going to get in the way over time, trying to >> continue to honor an interface that codifies internals at a certain >> point in time, and leaving them as-is will just continue to generate >> issues like the quoted bug. >> >> I don't "lean" on deprecation, I feel strongly that these should go away. > > I've worked on a deployment that uses them heavily and would be impacted > by their removal. They are a very convenient place to put code that > should run based on Nova events but I have yet to see a usage that > couldn't have been implemented by having a service listen to > notifications and run that same code. However there is no service that > does this. So the only argument I can see for keeping them is that it's > more convenient to put that code into Nova rather than implement > something that listens for notifications. And that's not a convincing > argument to me. The ability to inject files on a per-VM basis is one thing that would be missing. Right now decisions can be made inside the hook using attributes of the VM to decide whether and what to inject, including files generated inside the hook itself. I won't argue that the API is nice, it isn't, but I'd prefer a replace then deprecate model instead. I'm not entirely sure that the notifications contain all the same information as is available now in the hooks. rob > So I agree with moving forward on deprecation and think that > notifications provide a suitable replacement for the functionality > provided. > > >> >> --Dan >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] service metadata discovery URLs
If you hit the root page of many of the services (keystone, glance-api,
cinder and nova-api at least), the output will include the available
versions and their URLs.
These URLs are more or less hardcoded, with a config override.
In keystone these are public_endpoint and admin_endpoint, in glance-api
it is public_endpoint, etc.
This works fine for the default cases but falls down if you want to put
a proxy in front of it, I'm testing the TLS-proxy case. The advertised
values by default are unsecured which caused all sorts of interesting
failures in devstack.
These values are already available in the service catalog. I can fix
this short-term by setting values URLs in devstack but that is a hack at
best. It should either pull the values out of the catalog with each
request, or perhaps more performant, cache the values for some period.
Here is an example with the config options set in devstack with the
tls-proxy service enabled:
% curl https://localhost:5000
{versions: {values: [{status: stable, updated:
2015-03-30T00:00:00Z, media-types: [{base: application/json,
type: application/vnd.openstack.identity-v3+json}], id: v3.4,
links: [{href: https://localhost:5000/v3/;, rel: self}]},
{status: stable, updated: 2014-04-17T00:00:00Z, media-types:
[{base: application/json, type:
application/vnd.openstack.identity-v2.0+json}], id: v2.0, links:
[{href: https://localhost:5000/v2.0/;, rel: self}, {href:
http://docs.openstack.org/;, type: text/html, rel:
describedby}]}]}}
Here are two reviews demonstrating the issue that are blocked because
fixing it in the config is not the best way to fix this overall:
https://review.openstack.org/187346
https://review.openstack.org/188879
So in short, how should we address this? How will the fixes be
coordinated, as we don't need four or five different solutions to the
same problem?
rob
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [devstack] configuring https for glance client
Andrew Lazarev wrote: Hi Nova experts, Some time ago I figured out that devstack fails to stack with USE_SSL=True option because it doesn't configure nova to work with secured glace [1]. Support of secured glance was added to nova in Juno cycle [2], but it looks strange for me. Glance client takes settings form '[ssl]' section. The same section is used to set up nova server SSL settings. Other clients have separate sections in the config file (and switching to session use now), e.g. related code for cinder - [3]. I've created quick fix for the devstack - [4], but it would be nice to shed a light on nova plans around glance config before merging a workaround for devstack. So, the questions are: 1. Is it normal that glance client reads from '[ssl]' config section? 2. Is there a plan to move glance client to sessions use and move corresponding config section to '[glance]'? 3. Are any plans to run CI for USE_SSL=True use case? YES! It is quite a battle to keep SSL (native SSL and tls-proxy) working in devstack given the pace of development of the related projects. I've got a couple of reviews out now which may bring things to a state where the tempest smoke suite will pass in SSL mode. [1] - https://bugs.launchpad.net/devstack/+bug/1405484 I haven't seen that. What does your local.conf look like? Honestly I mostly test with the configuration that devstack-gate uses to have any chance of SSL being part of CI. Note too that I'm trying hard to be able to stop using service-specific CA options and instead add all CA certs to the global cert store and then things will just work. I originally start pushing for per-client SSL config sections and got pushback from a number of PTLs that didn't want their config options exploding. That leads to the python-requests problem. It includes a nifty function, where(), which returns the location of the global CA bundle. Packagers set this to match what is on individual platforms (Debian, Ubuntu Fedora/RHEL anyway). Unfortunately devstack often requires newer versions that ships with the OS's causing pip to install the upstream version, which of course defaults to using the bundle it ships, which ends up breaking lots of things. I'm stumped as to how to fix that. I made a ham-handed attempt at fixing it last year and was appropriately smacked down. rob __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] HTTPS client breaks nova
Flavio Percoco wrote: On 07/23/2014 06:05 PM, Rob Crittenden wrote: Rob Crittenden wrote: It looks like the switch to requests in python-glanceclient (https://review.openstack.org/#/c/78269/) has broken nova when SSL is enabled. I think it is related to the custom object that the glanceclient uses. If another connection gets pushed into the pool then things fail because the object isn't a glanceclient VerifiedHTTPSConnection object. The error seen is: 2014-07-22 16:20:57.571 ERROR nova.api.openstack req-e9a94169-9af4-45e8-ab95-1ccd3f8caf04 admin admin Caught error: VerifiedHTTPSConnection instance has no attribute 'insecure' What I see is that nova works until glance is invoked. These all work: $ nova flavor-list $ glance image-list $ nova net-list Now make it go boom: $ nova image-list ERROR (Unauthorized): Unauthorized (HTTP 401) (Request-ID: req-ee964e9a-c2a9-4be9-bd52-3f42c805cf2c) Now that a bad object is now in the pool nothing in nova works: $ nova list ERROR (Unauthorized): Unauthorized (HTTP 401) (Request-ID: req-f670db83-c830-4e75-b29f-44f61ae161a1) A restart of nova gets things back to normal. I'm working on enabling SSL everywhere (https://bugs.launchpad.net/devstack/+bug/1328226) either directly or using TLS proxies (stud). I'd like to eventually get SSL testing done as a gate job which will help catch issues like this in advance. rob FYI, my temporary workaround is to change the queue name (scheme) so the glance clients are handled separately: diff --git a/glanceclient/common/https.py b/glanceclient/common/https.py index 6416c19..72ed929 100644 --- a/glanceclient/common/https.py +++ b/glanceclient/common/https.py @@ -72,7 +72,7 @@ class HTTPSAdapter(adapters.HTTPAdapter): def __init__(self, *args, **kwargs): # NOTE(flaper87): This line forces poolmanager to use # glanceclient HTTPSConnection -poolmanager.pool_classes_by_scheme[https] = HTTPSConnectionPool +poolmanager.pool_classes_by_scheme[glance_https] = HTTPSConnectionPoo super(HTTPSAdapter, self).__init__(*args, **kwargs) def cert_verify(self, conn, url, verify, cert): @@ -92,7 +92,7 @@ class HTTPSConnectionPool(connectionpool.HTTPSConnectionPool): be used just when the user sets --no-ssl-compression. -scheme = 'https' +scheme = 'glance_https' def _new_conn(self): self.num_connections += 1 This at least lets me continue working. rob Hey Rob, Sorry for the late reply, I'll take a look into this. Ping, have you had a chance to look into it? thanks rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [glance] HTTPS client breaks nova
It looks like the switch to requests in python-glanceclient (https://review.openstack.org/#/c/78269/) has broken nova when SSL is enabled. I think it is related to the custom object that the glanceclient uses. If another connection gets pushed into the pool then things fail because the object isn't a glanceclient VerifiedHTTPSConnection object. The error seen is: 2014-07-22 16:20:57.571 ERROR nova.api.openstack req-e9a94169-9af4-45e8-ab95-1ccd3f8caf04 admin admin Caught error: VerifiedHTTPSConnection instance has no attribute 'insecure' What I see is that nova works until glance is invoked. These all work: $ nova flavor-list $ glance image-list $ nova net-list Now make it go boom: $ nova image-list ERROR (Unauthorized): Unauthorized (HTTP 401) (Request-ID: req-ee964e9a-c2a9-4be9-bd52-3f42c805cf2c) Now that a bad object is now in the pool nothing in nova works: $ nova list ERROR (Unauthorized): Unauthorized (HTTP 401) (Request-ID: req-f670db83-c830-4e75-b29f-44f61ae161a1) A restart of nova gets things back to normal. I'm working on enabling SSL everywhere (https://bugs.launchpad.net/devstack/+bug/1328226) either directly or using TLS proxies (stud). I'd like to eventually get SSL testing done as a gate job which will help catch issues like this in advance. rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance] HTTPS client breaks nova
Rob Crittenden wrote: It looks like the switch to requests in python-glanceclient (https://review.openstack.org/#/c/78269/) has broken nova when SSL is enabled. I think it is related to the custom object that the glanceclient uses. If another connection gets pushed into the pool then things fail because the object isn't a glanceclient VerifiedHTTPSConnection object. The error seen is: 2014-07-22 16:20:57.571 ERROR nova.api.openstack req-e9a94169-9af4-45e8-ab95-1ccd3f8caf04 admin admin Caught error: VerifiedHTTPSConnection instance has no attribute 'insecure' What I see is that nova works until glance is invoked. These all work: $ nova flavor-list $ glance image-list $ nova net-list Now make it go boom: $ nova image-list ERROR (Unauthorized): Unauthorized (HTTP 401) (Request-ID: req-ee964e9a-c2a9-4be9-bd52-3f42c805cf2c) Now that a bad object is now in the pool nothing in nova works: $ nova list ERROR (Unauthorized): Unauthorized (HTTP 401) (Request-ID: req-f670db83-c830-4e75-b29f-44f61ae161a1) A restart of nova gets things back to normal. I'm working on enabling SSL everywhere (https://bugs.launchpad.net/devstack/+bug/1328226) either directly or using TLS proxies (stud). I'd like to eventually get SSL testing done as a gate job which will help catch issues like this in advance. rob FYI, my temporary workaround is to change the queue name (scheme) so the glance clients are handled separately: diff --git a/glanceclient/common/https.py b/glanceclient/common/https.py index 6416c19..72ed929 100644 --- a/glanceclient/common/https.py +++ b/glanceclient/common/https.py @@ -72,7 +72,7 @@ class HTTPSAdapter(adapters.HTTPAdapter): def __init__(self, *args, **kwargs): # NOTE(flaper87): This line forces poolmanager to use # glanceclient HTTPSConnection -poolmanager.pool_classes_by_scheme[https] = HTTPSConnectionPool +poolmanager.pool_classes_by_scheme[glance_https] = HTTPSConnectionPoo super(HTTPSAdapter, self).__init__(*args, **kwargs) def cert_verify(self, conn, url, verify, cert): @@ -92,7 +92,7 @@ class HTTPSConnectionPool(connectionpool.HTTPSConnectionPool): be used just when the user sets --no-ssl-compression. -scheme = 'https' +scheme = 'glance_https' def _new_conn(self): self.num_connections += 1 This at least lets me continue working. rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] neutron config not working
Rob Crittenden wrote: Mark Kirkwood wrote: On 25/06/14 10:59, Rob Crittenden wrote: Before I get punted onto the operators list, I post this here because this is the default config and I'd expect the defaults to just work. Running devstack inside a VM with a single NIC configured and this in localrc: disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta enable_service neutron Q_USE_DEBUG_COMMAND=True Results in a successful install but no DHCP address assigned to hosts I launch and other oddities like no CIDR in nova net-list output. Is this still the default way to set things up for single node? It is according to https://wiki.openstack.org/wiki/NeutronDevstack That does look ok: I have an essentially equivalent local.conf: ... ENABLED_SERVICES+=,-n-net ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,q-metering,tempest I don't have 'neutron' specifically enabled... not sure if/why that might make any difference tho. However instance launching and ip address assignment seem to work ok. However I *have* seen the issue of instances not getting ip addresses in single host setups, and it is often due to use of virt io with bridges (with is the default I think). Try: nova.conf: ... libvirt_use_virtio_for_bridges=False Thanks for the suggestion. At least in master this was replaced by a new section, libvirt, but even setting it to False didn't do the trick for me. I see the same behavior. OK, I've tested the havana and icehouse branches in F-20 and they don't seem to have a working neutron either. I see the same thing. I can launch a VM but it isn't getting a DHCP address. Maybe I'll try in some Ubuntu release to see if this is Fedora-specific. rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] neutron config not working
Mark Kirkwood wrote: On 25/06/14 10:59, Rob Crittenden wrote: Before I get punted onto the operators list, I post this here because this is the default config and I'd expect the defaults to just work. Running devstack inside a VM with a single NIC configured and this in localrc: disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta enable_service neutron Q_USE_DEBUG_COMMAND=True Results in a successful install but no DHCP address assigned to hosts I launch and other oddities like no CIDR in nova net-list output. Is this still the default way to set things up for single node? It is according to https://wiki.openstack.org/wiki/NeutronDevstack That does look ok: I have an essentially equivalent local.conf: ... ENABLED_SERVICES+=,-n-net ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,q-metering,tempest I don't have 'neutron' specifically enabled... not sure if/why that might make any difference tho. However instance launching and ip address assignment seem to work ok. However I *have* seen the issue of instances not getting ip addresses in single host setups, and it is often due to use of virt io with bridges (with is the default I think). Try: nova.conf: ... libvirt_use_virtio_for_bridges=False Thanks for the suggestion. At least in master this was replaced by a new section, libvirt, but even setting it to False didn't do the trick for me. I see the same behavior. thanks rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [DevStack] neutron config not working
Before I get punted onto the operators list, I post this here because this is the default config and I'd expect the defaults to just work. Running devstack inside a VM with a single NIC configured and this in localrc: disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta enable_service neutron Q_USE_DEBUG_COMMAND=True Results in a successful install but no DHCP address assigned to hosts I launch and other oddities like no CIDR in nova net-list output. Is this still the default way to set things up for single node? It is according to https://wiki.openstack.org/wiki/NeutronDevstack thanks rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] SSL in Common client
Chmouel Boudjnah wrote: Rob Crittenden rcrit...@redhat.com writes: From what I found nothing has changed either upstream or in swift. If you are asking about the ability to disable SSL compression it is up to the OS to provide that so nothing was added when we changed swiftclient to requests. Most modern OSes have SSL compression by default, only Debian stable was still enabling it. The only feature that was left behind when we ported swiftclient to requests was the support of Expects (100-Continue) which is referenced upstream in this bug : https://github.com/kennethreitz/requests/issues/713 and it does not seem be trivial to add to requests Yes, that was my take as well. I didn't mean to come across as criticizing the swift client. I was just trying to outline the current state of things. rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] SSL in Common client
TL;DR
Work is happening on a unified client library. This provides the
opportunity to rework the way SSL options are handled. Can we discuss
this in one of the sessions at the Atlanta Summit in a few weeks?
https://blueprints.launchpad.net/oslo/+spec/common-client-library-2
outlines a path to a common client library in OpenStack. It is still a
work-in-progress though most projects have some amount included already
in openstack/common/apiclient
If you have some time and aren't familiar with this project, this thread
will bring you up to speed:
http://lists.openstack.org/pipermail/openstack-dev/2014-January/024441.html
Parallel to this effort was
https://wiki.openstack.org/wiki/SecureClientConnections which outlined
an effort to replace httplib{2} with requests which is mostly complete
with the exception of neutronclient and glanceclient which still use
httplib.
I'm trying to get devstack to the point where it can configure all the
services with SSL so it can be be part of the acceptance process. This
is for client communication, there is no expectation that anyone would
deploy native SSL endpoints. For the most part things just work. Most of
the issues I've run into are server to server communication relating to
passing in the CA certificate path.
This leads to two interrelated questions:
1. Given the common client, how much should be done in the interim to
clean things up?
2. How will configuration options be handled for server to server
communication?
Right now each project has its own local copy of the common client but
only exceptions are being used. Is there any guess on how soon the
common HTTP client can be in place? This may drive how much effort is
expended trying to clean up the existing client code.
There are significant, probably well-known differences between the
clients, and in the options available to clients used within several
servers to communicate as clients to other servers (e.g. nova to glance).
Here is a brief taste of what I'm talking about:
heatclient defines get_system_ca_file() which will use the system bundle
as a fallback. It is the only client project that does this.
Heat seems to have the most expansive set of configuration options,
providing a global clients section and service-specific
clients_service set of options. See
https://github.com/openstack/heat/blob/master/etc/heat/heat.conf.sample#L589
. It was suggested that this be considered for nova as well.
The nova team is working to consolidate the available CA options into a
single one in https://bugs.launchpad.net/nova/+bug/1299841 but leaves
behind the separate options for insecure connections.
Disabling SSL compression is important to swift and glance and their
clients provide a way to disable it, and each server that calls these
typically have configuration options to manage it (why there are even
options is unclear since the glance and swift server teams really want
this disabled).
How SSL is handled in the configuration files is separate from the
Common Client blueprint. Will there continue to be separate options for
insecure, ssl_compression and ca_certificates or will there be a single,
common option set somewhere, or a mixed bag?
How flexible do the CA certificates need to be? Given the distributed
nature of OpenStack, and the fact that some endpoints may be internal
and others external facing, it might seem reasonable to have separate
options for each service. It is also confusing and repetitive. The
heatclient get_system_ca_file() seems to be the way to go, along with an
override in the service configuration file.
That doesn't really handle compression or insecure though, which still
probably need to be per-service.
I think the heat/heatclient approach is the best, coupled with good
defaults in the common client. I think it should look like:
Common client:
ca_certificates = get_system_ca_file()
insecure = False
ssl_compression = False
Ideally the system CA is configured correctly so there is nothing to do
in the client except pass in a https endpoint.
Each server uses the heat method and defines:
[ client ] (most likely always empty)
[ client_service ]
I see are three possible options
ca_certificates_file = /path/to/file
insecure = Boolean
ssl_compression = Boolean
This is the most flexible setup but how one documents it without
confusing people I'm not entirely sure. I think the sample configuration
files should have the entire sections commented out, and heavily, that
these are for overrides only, and only when needed.
Backwards compatibility will be an issue, though the old options can be
deprecated and eventually removed. Or a script can probably be used to
migrate options to the new format.
rob
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] SSL in Common client
Dean Troyer wrote: On Fri, May 2, 2014 at 2:14 PM, Adam Young ayo...@redhat.com mailto:ayo...@redhat.com wrote: Did swift leave this behind when they switched to Requests? Swift and Glance clients were not changed to requests when I did the initial work in the fall of 2012 due to their use of chunked transfers. I've not really looked into this since then but do recall talk about being able to either implement the required changes in upstream requests or somehow hack it in from below. From what I found nothing has changed either upstream or in swift. rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] SSL in Common client
Dean Troyer wrote: On Fri, May 2, 2014 at 2:06 PM, Rob Crittenden rcrit...@redhat.com mailto:rcrit...@redhat.com wrote: I'm trying to get devstack to the point where it can configure all the services with SSL so it can be be part of the acceptance process. This is for client communication, there is no expectation that anyone would deploy native SSL endpoints. For the most part things just work. Most of the issues I've run into are server to server communication relating to passing in the CA certificate path. FWIW, DevStack has had the ability to do TLS termination using stud for all public API services, long before any of the individual service SSL/TLS configurations were usable. Using an external TLS termination solves the internal communication problem as long as internal services are configured properly. It also more closely matches what I have seen in real-world deployments. I'm not particularly worried about the endpoints. What I want to test are servers acting as clients and the CLI clients to secure endpoints. I want to ensure that SSL works for those cases where services are running on separate nodes, however they are secured (natively or with a proxy). It has been a while since I've tested this and it is likely to need some love. The basic structure, including building a root and intermediate CA to issue certs that look like real-world certs, has been present for almost a year and a half. I found the basic SSL code in pretty good shape so I suspect that it still works. rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
