Re: [openstack-dev] [TripleO][keystone] internal endpoints vs sanity

[Gyorgy Szombathelyi]

Hi,
> 
> 
> 
> On Mon, Jul 24, 2017 at 10:53 AM, Dmitry Tantsur   > wrote:
> 
> 
>   These questions are to the operators, and should be asked on
> openstack-operators IMO (maybe with tuning the overall tone to be a bit less
> aggressive).
> 
> 
> 
> So the question looks like this without tuning:
> 
>  - Do you think is it good idea to spam the users with internal data which
> useless for them unless they want to use it against you ?
> 

As a person who configured some OpenStack clusters, I can say that I use
the same name for internal and public endpoints, and configure the OpenStack
servers' host files to resolve this name to an internal address, where a load 
balancer
listens.
This is mainly because it makes TLS certificate handling much easier for us, we
can use a certificate for OpenStack services issued for one domain name only.
Second point of course, that there's no point to show the internal addresses to 
users
in the service catalog.

Br,
György
 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][keystone] internal endpoints vs sanity

[Attila Fazekas]

On Mon, Jul 24, 2017 at 10:53 AM, Dmitry Tantsur 
wrote:

> These questions are to the operators, and should be asked on
> openstack-operators IMO (maybe with tuning the overall tone to be a bit
> less aggressive).
>

So the question looks like this without tuning:
 - Do you think is it good idea to spam the users with internal data which
useless for them unless they want to use it against you ?


>
> On 07/24/2017 10:23 AM, Attila Fazekas wrote:
>
>> Thanks for your answer.
>>
>> The real question is do we agree in the
>> internalULR usage what suggested in [1] is a bad security practice
>> and should not be told to operators at all.
>>
>> Also we should try to get rid off the enpointTypes in keystone v4.
>>
>
> Let's not seriously talk about keystone v4 at this point, we haven't
> gotten rid of v2 so far.
>
> Eventually it will come, but until that the story told to operators could
be we are going to remove the
interfaces (admin/internal/public) from the keystone catalog.


>
>> Do we have any good (not just making happy funny dev envs) to keep
>> endpoint types ?
>>
>
> I suspect any external SSL termination proxy. And anything else that will
> make the URLs exposed to end users look different from ones exposed to
> services.
>

The only real question is how many people would mind to use SSL/TLS also
internally
across the  services, when https is the one provided to the end-users.

It does not means the LB to backend needs to be SSL, it can still remain
HTTP regardless to the catalog entry.

If the internal both-way no-SSL communication is really important for the
deployers and we do not
want to change how the keystone API behave we might put
the service urls next to to auth urls in the  keystone_authtoken_section
kind of sections .

Many service has multiple  keystone_authtoken_section [3],
but for example `heat` does not have dedicated auth like section for all
related services.

The options available keystoneauth1 is usually directly exposed to the
service config,
so introducing a `catalog_override` option which accepts a json file can be
the simplest option.

Again, it is only required if you really want to use different protocol
internally than  for the public.
This should not be in a security best practice guide either, but if there
is a real user request for it so be it.


>
> Speaking of DNS, I also suspect there may be a micro-optimization in not
> making the services use it when talking to each other, while still
> providing names to end users.
>
>

If we are speaking about micro optimization, the above way would
open up the way to have services to choose `same host`, `same segment`
service instances
when it makes sense (usually does not).

Most of the networking libraries has built in intelligence to cache DNS
responses,
the dns lookup typically causes extra <0.1ms latency an openstack service
frequently needs
more than 5 ms to respond.

But if you really want to do some micro optimization here,
there are multiple small dns services available which can run on the
localhost and provide faster response
then a remote one and they are also able to hide dns infrastructure
downtimes.

The devstack vms are using unbound for DNS caching.

As always, you can use /etc/hosts file to bypass the DNS lookups,
however the /etc/hosts not expected to do Round Robin, but if you were
happy without DNS
 you will not notice it.

nscd might have surprising changing behavior, but available in all Linux
distros,
likely you want to decrease the negative-time-to-live times in most cases.


[3]
https://docs.openstack.org/ocata/config-reference/shared-file-systems/samples/manila.conf.html


>
>>
>>
>> On Fri, Jul 21, 2017 at 1:37 PM, Giulio Fidente > > wrote:
>>
>> Only a comment about the status in TripleO
>>
>> On 07/21/2017 12:40 PM, Attila Fazekas wrote:
>>
>> [...]
>>
>> > We should seriously consider using names instead of ip address also
>> > on the devstack gates to avoid people thinking the catalog entries
>> > meant to be used with ip address and keystone is a replacement for
>> DNS.
>>
>> this is configurable, you can have names or ips in the keystone
>> endpoints ... actually you can chose to use names or ips independently
>> for each service and even for the different endpoints
>> (Internal/Admin/Public) of the same service
>>
>> if an operator, like you suggested, configures the DNS to resolve
>> different IPs for the same name basing on where the request comes
>> from,
>> then he can use the same 'hostname' for all Public, Admin and Internal
>> endpoints which I *think* is what you're suggesting
>>
>> also using names is the default when ssl is enabled
>>
>> check environments/ssl/tls-endpoints-public-dns.yaml and note how
>> EndpointMap can resolve to CLOUDNAME or IP_ADDRESS
>>
>> adding Juan on CC as he did a great work around this and can help
>> further
>> 

Re: [openstack-dev] [TripleO][keystone] internal endpoints vs sanity

[Attila Fazekas]

>> "While it may not be a setup that everyone wants, for some deployers
having a public and internal is important."
To be more precise:
In some deployment based on in the initial directives the operators run
into an issue where the `public and internal named urls in keystone` looked
like a possible solution.

>> "Those deployers do not seem to mind the RFC1918 showing up in the
catalog, "

These deployers either just showing the public interfaces just to the
trusted administrators(/power users) or not aware of the possible risks.


>> "if they're doing point-to-point firewalling (as they should be) the
private addresses should not be considered 'secret'  "

Based on your trust in your network setup , you should also consider
removing the authentication from (for example) mariadb,
because if you really think your services are safe from the public you do
not need it either.

A random example for why this address leakage can be dangerous.:
 - Based on your private address the attacker is able to figure out (or
just better guess) where is your non-openstack openstack backend services
 - One of your backend services has weak or no authentication
 - You have an openstack service which is able to connect to arbitrary
address on user request (connection to the backend is explicitly allowed by
firewall)
 ---> possible one hit exploitation

If you think these were never was in an OpenStack together, think again and
read the CVEs and the
deployment guides and script.

We must not make the task for the crackers easier by exposing internal
information.
The addresses are frequently not dangerous alone, but in an OpenStack sized
thing it
can become very dangerous together with another `minor` issues.

Another randomly picked issue regarding to an internal url expose:
 1. have service which has some internal info which would be useful for
another service
 2. expose the internal data to all users on the API
 3. have different backend where the same filed is confidential by nature

We will run into similar issue again and again if we are not strict about
not
exposing internal info.

Whatever you think you are solving by internal urls,
can be solved in multiple other ways without leaking information, in most
cases
we also does not need to modify an OpenStack service code for solving it.

Because the internal urls are useless for the unprivileged users, they
should
not receive them at all, even tough we might have users who simply do not
care
about this, they will not die if we move to more secure solution.


>
> On Fri, Jul 21, 2017 at 1:37 PM, Giulio Fidente > > wrote:
>>
>> Only a comment about the status in TripleO
>>
>> On 07/21/2017 12:40 PM, Attila Fazekas wrote:
>>
>> [...]
>>
>> > We should seriously consider using names instead of ip address also
>> > on the devstack gates to avoid people thinking the catalog entries
>> > meant to be used with ip address and keystone is a replacement for
>> DNS.
>>
>> this is configurable, you can have names or ips in the keystone
>> endpoints ... actually you can chose to use names or ips independently
>> for each service and even for the different endpoints
>> (Internal/Admin/Public) of the same service
>>
>> if an operator, like you suggested, configures the DNS to resolve
>> different IPs for the same name basing on where the request comes
>> from,
>> then he can use the same 'hostname' for all Public, Admin and Internal
>> endpoints which I *think* is what you're suggesting
>>
>> also using names is the default when ssl is enabled
>>
>> check environments/ssl/tls-endpoints-public-dns.yaml and note how
>> EndpointMap can resolve to CLOUDNAME or IP_ADDRESS
>>
>> adding Juan on CC as he did a great work around this and can help
>> further
>> --
>> Giulio Fidente
>> GPG KEY: 08D733BA
>>
>>
>>
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][keystone] internal endpoints vs sanity

[Monty Taylor]

On 07/24/2017 04:23 PM, Attila Fazekas wrote:

Thanks for your answer.

The real question is do we agree in the
internalULR usage what suggested in [1] is a bad security practice
and should not be told to operators at all.

Also we should try to get rid off the enpointTypes in keystone v4.

Do we have any good (not just making happy funny dev envs) to keep
endpoint types ?


First step is stopping to use the word "endpoint_type" - it's 
"interface". Also, "internalURL" is legacy keystone v2, it's "internal"


"admin" interface is a thing that people shouldn't use. It's a holdover 
from a day when keystone did weird things. Nobody else should use it ever.


However, we just ADDED the ability to more intelligently consume 
interface so that public/internal cases can be handled better based on 
use-cases from deployers, so I do not think they're going to go away any 
time in the forseeable future.


For instance, we added the ability to pass a list of interface values to 
keystoneauth's endpoint_filter - and these are also now in the adapter 
options so that the default value for "interface" for nova talking to 
neutron can be ['internal', 'public'] - which says: "please default to 
using the internal interface if one exists, otherwise fall back to the 
public interface"


While it may not be a setup that everyone wants, for some deployers 
having a public and internal is important. I know several clouds have 
deployed completely separate API tiers and registered them as "internal" 
so that they could be assured that service-to-service communications 
worked well even if end-users were hammering the public endpoints. Those 
deployers do not seem to mind the RFC1918 showing up in the catalog, and 
if they're doing point-to-point firewalling (as they should be) the 
private addresses should not be considered 'secret' so there's no real 
problem exposing them in the catalog.



On Fri, Jul 21, 2017 at 1:37 PM, Giulio Fidente > wrote:


Only a comment about the status in TripleO

On 07/21/2017 12:40 PM, Attila Fazekas wrote:

[...]

> We should seriously consider using names instead of ip address also
> on the devstack gates to avoid people thinking the catalog entries
> meant to be used with ip address and keystone is a replacement for DNS.

this is configurable, you can have names or ips in the keystone
endpoints ... actually you can chose to use names or ips independently
for each service and even for the different endpoints
(Internal/Admin/Public) of the same service

if an operator, like you suggested, configures the DNS to resolve
different IPs for the same name basing on where the request comes from,
then he can use the same 'hostname' for all Public, Admin and Internal
endpoints which I *think* is what you're suggesting

also using names is the default when ssl is enabled

check environments/ssl/tls-endpoints-public-dns.yaml and note how
EndpointMap can resolve to CLOUDNAME or IP_ADDRESS

adding Juan on CC as he did a great work around this and can help
further
--
Giulio Fidente
GPG KEY: 08D733BA




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][keystone] internal endpoints vs sanity

[Dmitry Tantsur]

These questions are to the operators, and should be asked on openstack-operators 
IMO (maybe with tuning the overall tone to be a bit less aggressive).


On 07/24/2017 10:23 AM, Attila Fazekas wrote:

Thanks for your answer.

The real question is do we agree in the
internalULR usage what suggested in [1] is a bad security practice
and should not be told to operators at all.

Also we should try to get rid off the enpointTypes in keystone v4.


Let's not seriously talk about keystone v4 at this point, we haven't gotten rid 
of v2 so far.




Do we have any good (not just making happy funny dev envs) to keep
endpoint types ?


I suspect any external SSL termination proxy. And anything else that will make 
the URLs exposed to end users look different from ones exposed to services.


Speaking of DNS, I also suspect there may be a micro-optimization in not making 
the services use it when talking to each other, while still providing names to 
end users.






On Fri, Jul 21, 2017 at 1:37 PM, Giulio Fidente > wrote:


Only a comment about the status in TripleO

On 07/21/2017 12:40 PM, Attila Fazekas wrote:

[...]

> We should seriously consider using names instead of ip address also
> on the devstack gates to avoid people thinking the catalog entries
> meant to be used with ip address and keystone is a replacement for DNS.

this is configurable, you can have names or ips in the keystone
endpoints ... actually you can chose to use names or ips independently
for each service and even for the different endpoints
(Internal/Admin/Public) of the same service

if an operator, like you suggested, configures the DNS to resolve
different IPs for the same name basing on where the request comes from,
then he can use the same 'hostname' for all Public, Admin and Internal
endpoints which I *think* is what you're suggesting

also using names is the default when ssl is enabled

check environments/ssl/tls-endpoints-public-dns.yaml and note how
EndpointMap can resolve to CLOUDNAME or IP_ADDRESS

adding Juan on CC as he did a great work around this and can help further
--
Giulio Fidente
GPG KEY: 08D733BA




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][keystone] internal endpoints vs sanity

[Attila Fazekas]

Thanks for your answer.

The real question is do we agree in the
internalULR usage what suggested in [1] is a bad security practice
and should not be told to operators at all.

Also we should try to get rid off the enpointTypes in keystone v4.

Do we have any good (not just making happy funny dev envs) to keep
endpoint types ?



On Fri, Jul 21, 2017 at 1:37 PM, Giulio Fidente  wrote:

> Only a comment about the status in TripleO
>
> On 07/21/2017 12:40 PM, Attila Fazekas wrote:
>
> [...]
>
> > We should seriously consider using names instead of ip address also
> > on the devstack gates to avoid people thinking the catalog entries
> > meant to be used with ip address and keystone is a replacement for DNS.
>
> this is configurable, you can have names or ips in the keystone
> endpoints ... actually you can chose to use names or ips independently
> for each service and even for the different endpoints
> (Internal/Admin/Public) of the same service
>
> if an operator, like you suggested, configures the DNS to resolve
> different IPs for the same name basing on where the request comes from,
> then he can use the same 'hostname' for all Public, Admin and Internal
> endpoints which I *think* is what you're suggesting
>
> also using names is the default when ssl is enabled
>
> check environments/ssl/tls-endpoints-public-dns.yaml and note how
> EndpointMap can resolve to CLOUDNAME or IP_ADDRESS
>
> adding Juan on CC as he did a great work around this and can help further
> --
> Giulio Fidente
> GPG KEY: 08D733BA
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][keystone] internal endpoints vs sanity

[Giulio Fidente]

Only a comment about the status in TripleO

On 07/21/2017 12:40 PM, Attila Fazekas wrote:

[...]

> We should seriously consider using names instead of ip address also
> on the devstack gates to avoid people thinking the catalog entries
> meant to be used with ip address and keystone is a replacement for DNS.

this is configurable, you can have names or ips in the keystone
endpoints ... actually you can chose to use names or ips independently
for each service and even for the different endpoints
(Internal/Admin/Public) of the same service

if an operator, like you suggested, configures the DNS to resolve
different IPs for the same name basing on where the request comes from,
then he can use the same 'hostname' for all Public, Admin and Internal
endpoints which I *think* is what you're suggesting

also using names is the default when ssl is enabled

check environments/ssl/tls-endpoints-public-dns.yaml and note how
EndpointMap can resolve to CLOUDNAME or IP_ADDRESS

adding Juan on CC as he did a great work around this and can help further
-- 
Giulio Fidente
GPG KEY: 08D733BA

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO][keystone] internal endpoints vs sanity

[Attila Fazekas]

Hi All,

I thought it is already well know fact the endpoint types are there ONLY
for historical reasons, today they just exists to confuse the one who tries
to deploy OpenStack,
but it is considered as a deprecated concept and it will die out sooner or
later.

The keystone v3 API already allows to not define internal or admin
endpoints at all.

I just noticed the current documentation encourages the internal endpoint
usage. [1]

Is there anybody here who thinks it is a great idea to show private address
to the end users ?
Even tough some people might consider this cwe-200, but I hope at least
looks bad to everyone.

The internal endpoints should not be used for telling internal information
to the
OpenStack services itself.  We are not putting mariadb and rabbitmq address
to the catalog as well, we have config files for that.

Ideally the end users should not even know we are using different network
paths or not,
so the internalURL entries should not be different addresses than the
public one
or they should not be defined at all.

I hope nobody really thinks the public catalog entries expected to contain
ip address instead
 of domain names by any best practice guide.

We are just using ip address in the catalog for dev/test environment,
but  in an ideal case the identity url should start with https:// ,
and it should continue with a domain name, which have several A and 
entry
and the certificate wound not be for a self signed private ip address.

Is there anybody who really thinks we are putting  http:///..
into the catalog on the gate because it is the best practice ?

You can configure your DNS server properly [2] or use the /etc/hosts file,
when for some reason you want some nodes to use different ip address
for reaching the OpenStack services.

Keystone does not needs to solve anything there,
these issues are solved decodes before OpenStack even existed.

I cannot take the single internalURL usage as a serious response for
`isolated networks` ,
because it does not scales when you want divide your network even more.
Adding internal2URL, internal3URL is not a great idea either.

We should seriously consider using names instead of ip address also
on the devstack gates to avoid people thinking the catalog entries
meant to be used with ip address and keystone is a replacement for DNS.

Using https likely a bad idea in a regular dev environment,
but I hope we agree sending unencrypted credentials over the wire
is not a recommended best practice.

Best Regards,
Attila


[1]
https://docs.openstack.org/security-guide/api-endpoints/api-endpoint-configuration-recommendations.html

[2]
https://serverfault.com/questions/332440/dns-bind-how-to-return-a-different-ip-based-on-requests-subnet
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev