Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing

[Flavio Percoco]

On 22/03/16 17:20 -0400, Adam Young wrote:

On 03/22/2016 09:15 AM, Flavio Percoco wrote:

   On 21/03/16 21:43 -0400, Adam Young wrote:

   I had a good discussion with the Nova folks in IRC today.

   My goal was to understand what could talk to what, and the short
   according to dansmith

   " any node in nova land has to be able to talk to the queue for any
   other one for the most part: compute->compute, compute->conductor,
   conductor->compute, api->everything. There might be a few exceptions,
   but not worth it, IMHO, in the current architecture."

   Longer conversation is here:
   http://eavesdrop.openstack.org/irclogs/%23openstack-nova/
   %23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27

   Right now, the message queue is a nightmare.  All sorts of sensitive
   information flows over the message queue: Tokens (including admin) are
   the most obvious.  Every piece of audit data. All notifications and all
   control messages.

   Before we continue down the path of "anything can talk to anything" can
   we please map out what needs to talk to what, and why?  Many of the use
   cases seem to be based on something that should be kicked off by the
   conductor, such as "migrate, resize, live-migrate" and it sounds like
   there are plans to make that happen.

   So, let's assume we can get to the point where, if node 1 needs to talk
   to node 2, it will do so only via the conductor.  With that in place,
   we can put an access control rule in place:


   I don't think this is going to scale well. Eventually, this will require
   evolving the conductor to some sort of message scheduler, which is pretty
   much
   what the message bus is supposed to do.


I'll limit this to what happens with Rabbit and QPID (AMQP1.0) and leave 0 our
of it for now.  I'll use rabbit as shorthand for both these, but the rules are
the same for qpid.


Sorry for the pedantic nitpick but, it's not Qpid. I'm afraid calling it Qpid
will just confuse people on what we're really talking about here. The amqp1
driver is based on the AMQP 1.0 protocol, which is brokerless. The library used
in oslo.messaging is qpid-proton (A.K.A Proton). Qpid is just the name of the
Apache Foundation family that these projects belong to (including Qpidd the old
broker which we don't support anymore in oslo.messaging).



For, say, a migrate operation, the call goes to API, controller, and eventually
down to one of the compute nodes.  Source? Target?  I don't know the code well
enough to say, but let's say it is the source.  It sends an RPC message to the
target node.  The message goes to  the central broker right now, and then back
down to the targen node.  Meanwhile, the source node has set up a reply queue
and that queue name has gone into the message.  The target machine responds  by
getting a reference to the response queue and sends a message.  This message
goes up to the broker, and then down to the the source node.

A man in the middle could sit there and also read off the queue. It could
modify a message, with its own response queue, and happily tranfer things back
and forth.

So, we have the HMAC proposal, which then puts crypto and key distribution all
over the place.  Yes, it would guard against a MITM attack, but the cost in
complexity and processor time it high.


Rabbit does not have a very flexible ACL scheme, bascially, a RegEx per Rabbit
user.  However, we could easily spin up a new queue for direct node to node
communication that did meet an ACL regex.  For example, if we said that the
regex was that the node could only read/write queues that have its name in
them, to make a request and response queue between node-1 and node-2 we could
create a queues


node-1-node-2
node-1-node-2--reply


So, instead of a single queue request, there are two.  And conductor could tell
the target node: start listening on this queue.


Or, we could pass the message through the conductor.  The request message goes
from node-1 to conductor,  where conductor validates the businees logic of the
message, then puts it into the message queue for node-2.  Responses can then go
directly back from node-2 to node-1 the way they do now.

OR...we could set up a direct socket between the two nodes, with the socket set
up info going over the broker.  OR we could use a web server,  OR send it over
SNMP.  Or SMTP, OR TFTP.  There are many ways to get the messages from node to
node.

If  we are going to use the message broker to do this, we should at least make
it possible to secure it, even if it is not the default approach.

It might be possible to use a broker specific technology to optimize this, but
I am not a Rabbit expert.  Maybe there is some way of filtering messages?




   1.  Compute nodes can only read from the queue compute.
   -novacompute-.localdomain
   2.  Compute nodes can only write to response queues in the RPC vhost
   3.  

Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing

[Adam Young]

On 03/22/2016 05:42 PM, Dan Smith wrote:

Shouldn't we be trying to remove central bottlenecks by
decentralizing communications where we can?

I think that's a good goal to continue having. Some deployers have
setup firewalls between compute nodes, or between compute nodes and
the database, so we use the conductor to facilitate communications
between those nodes. But in general we don't want to send all
communications through the conductor.

Yep, I think we generally look forward to having all the resize and
migrate communication coordinated through conductor, but not really for
security reasons specifically. However, I don't think that pumping
everything through conductor for, say, api->compute communication is
something we should do.


So, Api to compute is probably fine as is.  I assume that most of that 
goes in the same queue as the conductor uses.


This assumes that we equally trust conductor and the API server, but I 
think if either is compromised, all bets are off anyway.




As several of us said in IRC yesterday, I'd really like nodes to be able
to authenticate the sender of a message and not do things based on who
sent it and whether that makes sense or not.
I read that as "we want to do HMAC outside of the Queue" and,as I said 
before, we tried that.  No one picked it up, Key distribution is a 
nightmare, and unless you do asymmetric cryptography, you need to have a 
separate shared secret for each reader and writer:  there is no pub-sub 
with symmetric crypto.


And we should not be rolling our own security.


  Adding a bunch of
broker-specific configuration requirements to achieve a security goal
(and thus assuming the queue is never compromised) is not really where I
want to see us go.


Nothing here is broker specific.  The rules are the same for Rabbit, 
QPID and 0MQ.


Message Brokers are a key piece of technology in a lot of enterprise 
software. It is possible to secure them.  Denying the operators the 
ability to secure them because we don't trust the brokers is not fair to 
the operators.




--Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing

[Dan Smith]

>> Shouldn't we be trying to remove central bottlenecks by
>> decentralizing communications where we can?
> 
> I think that's a good goal to continue having. Some deployers have
> setup firewalls between compute nodes, or between compute nodes and
> the database, so we use the conductor to facilitate communications
> between those nodes. But in general we don't want to send all
> communications through the conductor.

Yep, I think we generally look forward to having all the resize and
migrate communication coordinated through conductor, but not really for
security reasons specifically. However, I don't think that pumping
everything through conductor for, say, api->compute communication is
something we should do.

As several of us said in IRC yesterday, I'd really like nodes to be able
to authenticate the sender of a message and not do things based on who
sent it and whether that makes sense or not. Adding a bunch of
broker-specific configuration requirements to achieve a security goal
(and thus assuming the queue is never compromised) is not really where I
want to see us go.

--Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing

[Adam Young]

On 03/22/2016 09:15 AM, Flavio Percoco wrote:

On 21/03/16 21:43 -0400, Adam Young wrote:

I had a good discussion with the Nova folks in IRC today.

My goal was to understand what could talk to what, and the short 
according to dansmith


" any node in nova land has to be able to talk to the queue for any 
other one for the most part: compute->compute, compute->conductor, 
conductor->compute, api->everything. There might be a few exceptions, 
but not worth it, IMHO, in the current architecture."


Longer conversation is here:
http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27 



Right now, the message queue is a nightmare.  All sorts of sensitive 
information flows over the message queue: Tokens (including admin) 
are the most obvious.  Every piece of audit data. All notifications 
and all control messages.


Before we continue down the path of "anything can talk to anything" 
can we please map out what needs to talk to what, and why?  Many of 
the use cases seem to be based on something that should be kicked off 
by the conductor, such as "migrate, resize, live-migrate" and it 
sounds like there are plans to make that happen.


So, let's assume we can get to the point where, if node 1 needs to 
talk to node 2, it will do so only via the conductor.  With that in 
place, we can put an access control rule in place:


I don't think this is going to scale well. Eventually, this will require
evolving the conductor to some sort of message scheduler, which is 
pretty much

what the message bus is supposed to do.


I'll limit this to what happens with Rabbit and QPID (AMQP1.0) and leave 
0 our of it for now.  I'll use rabbit as shorthand for both these, but 
the rules are the same for qpid.




For, say, a migrate operation, the call goes to API, controller, and 
eventually down to one of the compute nodes.  Source? Target?  I don't 
know the code well enough to say, but let's say it is the source.  It 
sends an RPC message to the target node.  The message goes to  the 
central broker right now, and then back down to the targen node.  
Meanwhile, the source node has set up a reply queue and that queue name 
has gone into the message.  The target machine responds  by getting a 
reference to the response queue and sends a message.  This message goes 
up to the broker, and then down to the the source node.


A man in the middle could sit there and also read off the queue. It 
could modify a message, with its own response queue, and happily tranfer 
things back and forth.


So, we have the HMAC proposal, which then puts crypto and key 
distribution all over the place.  Yes, it would guard against a MITM 
attack, but the cost in complexity and processor time it high.



Rabbit does not have a very flexible ACL scheme, bascially, a RegEx per 
Rabbit user.  However, we could easily spin up a new queue for direct 
node to node communication that did meet an ACL regex.  For example, if 
we said that the regex was that the node could only read/write queues 
that have its name in them, to make a request and response queue between 
node-1 and node-2 we could create a queues



node-1-node-2
node-1-node-2--reply


So, instead of a single queue request, there are two.  And conductor 
could tell the target node: start listening on this queue.



Or, we could pass the message through the conductor.  The request 
message goes from node-1 to conductor,  where conductor validates the 
businees logic of the message, then puts it into the message queue for 
node-2.  Responses can then go directly back from node-2 to node-1 the 
way they do now.


OR...we could set up a direct socket between the two nodes, with the 
socket set up info going over the broker.  OR we could use a web 
server,  OR send it over SNMP.  Or SMTP, OR TFTP.  There are many ways 
to get the messages from node to node.


If  we are going to use the message broker to do this, we should at 
least make it possible to secure it, even if it is not the default approach.


It might be possible to use a broker specific technology to optimize 
this, but I am not a Rabbit expert.  Maybe there is some way of 
filtering messages?





1.  Compute nodes can only read from the queue 
compute.-novacompute-.localdomain

2.  Compute nodes can only write to response queues in the RPC vhost
3.  Compute nodes can only write to notification queus in the 
notification host.


I know that with AMQP, we should be able to identify the writer of a 
message.  This means that each compute node should have its own 
user.  I have identified how to do that for Rabbit and QPid.  I 
assume for 0mq is would make sense to use ZAP 
(http://rfc.zeromq.org/spec:27) but I'd rather the 0mq maintainers 
chime in here.




NOTE: Gentle reminder that qpidd has been removed from oslo.messaging.


Yes, but QPID is proton is AMQP1.0 and I did a proof of concept with it 
last summer.  It supports encryption and authentication over GSSAPI and 

Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing

[Flavio Percoco]

On 21/03/16 21:43 -0400, Adam Young wrote:

I had a good discussion with the Nova folks in IRC today.

My goal was to understand what could talk to what, and the short 
according to dansmith


" any node in nova land has to be able to talk to the queue for any 
other one for the most part: compute->compute, compute->conductor, 
conductor->compute, api->everything. There might be a few exceptions, 
but not worth it, IMHO, in the current architecture."


Longer conversation is here:
http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27

Right now, the message queue is a nightmare.  All sorts of sensitive 
information flows over the message queue: Tokens (including admin) are 
the most obvious.  Every piece of audit data. All notifications and 
all control messages.


Before we continue down the path of "anything can talk to anything" 
can we please map out what needs to talk to what, and why?  Many of 
the use cases seem to be based on something that should be kicked off 
by the conductor, such as "migrate, resize, live-migrate" and it 
sounds like there are plans to make that happen.


So, let's assume we can get to the point where, if node 1 needs to 
talk to node 2, it will do so only via the conductor.  With that in 
place, we can put an access control rule in place:


I don't think this is going to scale well. Eventually, this will require
evolving the conductor to some sort of message scheduler, which is pretty much
what the message bus is supposed to do.

1.  Compute nodes can only read from the queue 
compute.-novacompute-.localdomain

2.  Compute nodes can only write to response queues in the RPC vhost
3.  Compute nodes can only write to notification queus in the 
notification host.


I know that with AMQP, we should be able to identify the writer of a 
message.  This means that each compute node should have its own user.  
I have identified how to do that for Rabbit and QPid.  I assume for 
0mq is would make sense to use ZAP (http://rfc.zeromq.org/spec:27) but 
I'd rather the 0mq maintainers chime in here.




NOTE: Gentle reminder that qpidd has been removed from oslo.messaging.

I think you can configure rabbit, amqp1 and other technologies to do what you're
suggesting here without much trouble. TBH, I'm not sure how many chances would
be required in Nova (or even oslo.messaging) but I'd dare to say non are
required.

I think it is safe (and sane) to have the same use on the compute node 
communicate with  Neutron, Nova, and Ceilometer.  This will avoid a 
false sense of security: if one is compromised, they are all going to 
be compromised.  Plan accordingly.


Beyond that, we should have message broker users for each of the 
components that is a client of the broker.


Applications that run on top of the cloud, and that do not get 
presence on the compute nodes, should have their own VHost.  I see 
Sahara on my Tripleo deploy, but I assume there are others.  Either 
they completely get their own vhost, or the apps should share one 
separate from the RPC/Notification vhosts we currently have.  Even 
Heat might fall into this category.


Note that those application users can be allowed to read from the 
notification queues if necessary.  They just should not be using the 
same vhost for their own traffic.


Please tell me if/where I am blindingly wrong in my analysis.



I guess my question is: Have you identified things that need to be changed in
any of the projects for this to be possible? Or is it a pure deployment
recommendation/decision?

I'd argue that any change (assuming changes are required) are likely to happen
in specific projects (Nova, Neutron, etc) and that once this scenario is
supported, it'll remain a deployment choice to follow it or not. If I want my
undercloud services to use a single vhost and a single user, I must be able to
do that. The proposal in this email complicates deployments significantly,
despite it making sense from a security stand point.

One more thing. Depending on the messaging technology, having different virtual
hosts may have an impact on the performance when running under huge loads given
the fact that the data will be partitioned differently and, therefore,
written/read differently. I don't have good data at hand about this, sorry.

Flavio

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing

[Andrew Laski]

On Tue, Mar 22, 2016, at 07:00 AM, Doug Hellmann wrote:
> 
> 
> > On Mar 21, 2016, at 9:43 PM, Adam Young  wrote:
> > 
> > I had a good discussion with the Nova folks in IRC today.
> > 
> > My goal was to understand what could talk to what, and the short according 
> > to dansmith
> > 
> > " any node in nova land has to be able to talk to the queue for any other 
> > one for the most part: compute->compute, compute->conductor, 
> > conductor->compute, api->everything. There might be a few exceptions, but 
> > not worth it, IMHO, in the current architecture."
> > 
> > Longer conversation is here:
> > http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27
> > 
> > Right now, the message queue is a nightmare.  All sorts of sensitive 
> > information flows over the message queue: Tokens (including admin) are the 
> > most obvious.  Every piece of audit data. All notifications and all control 
> > messages.
> > 
> > Before we continue down the path of "anything can talk to anything" can we 
> > please map out what needs to talk to what, and why?  Many of the use cases 
> > seem to be based on something that should be kicked off by the conductor, 
> > such as "migrate, resize, live-migrate" and it sounds like there are plans 
> > to make that happen.
> > 
> > So, let's assume we can get to the point where, if node 1 needs to talk to 
> > node 2, it will do so only via the conductor.  With that in place, we can 
> > put an access control rule in place:

If you specifically mean compute nodes by "node 1" and "node 2" that is
something that's being worked towards. If you mean "node" more generally
that's not something that is planned.

> 
> Shouldn't we be trying to remove central bottlenecks by decentralizing
> communications where we can?

I think that's a good goal to continue having. Some deployers have setup
firewalls between compute nodes, or between compute nodes and the
database, so we use the conductor to facilitate communications between
those nodes. But in general we don't want to send all communications
through the conductor. 

> 
> Doug
> 
> 
> > 
> > 1.  Compute nodes can only read from the queue 
> > compute.-novacompute-.localdomain
> > 2.  Compute nodes can only write to response queues in the RPC vhost
> > 3.  Compute nodes can only write to notification queus in the notification 
> > host.
> > 
> > I know that with AMQP, we should be able to identify the writer of a 
> > message.  This means that each compute node should have its own user.  I 
> > have identified how to do that for Rabbit and QPid.  I assume for 0mq is 
> > would make sense to use ZAP (http://rfc.zeromq.org/spec:27) but I'd rather 
> > the 0mq maintainers chime in here.
> > 
> > I think it is safe (and sane) to have the same use on the compute node 
> > communicate with  Neutron, Nova, and Ceilometer.  This will avoid a false 
> > sense of security: if one is compromised, they are all going to be 
> > compromised.  Plan accordingly.
> > 
> > Beyond that, we should have message broker users for each of the components 
> > that is a client of the broker.
> > 
> > Applications that run on top of the cloud, and that do not get presence on 
> > the compute nodes, should have their own VHost.  I see Sahara on my Tripleo 
> > deploy, but I assume there are others.  Either they completely get their 
> > own vhost, or the apps should share one separate from the RPC/Notification 
> > vhosts we currently have.  Even Heat might fall into this category.
> > 
> > Note that those application users can be allowed to read from the 
> > notification queues if necessary.  They just should not be using the same 
> > vhost for their own traffic.
> > 
> > Please tell me if/where I am blindingly wrong in my analysis.
> > 
> > 
> > 
> > 
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing

[gordon chung]

On 21/03/2016 9:43 PM, Adam Young wrote:
> I had a good discussion with the Nova folks in IRC today.
>
> My goal was to understand what could talk to what, and the short
> according to dansmith
>
> " any node in nova land has to be able to talk to the queue for any
> other one for the most part: compute->compute, compute->conductor,
> conductor->compute, api->everything. There might be a few exceptions,
> but not worth it, IMHO, in the current architecture."
>
> Longer conversation is here:
>   
> http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27
>
> Right now, the message queue is a nightmare.  All sorts of sensitive
> information flows over the message queue: Tokens (including admin) are
> the most obvious.  Every piece of audit data. All notifications and all
> control messages.
>
> Before we continue down the path of "anything can talk to anything" can
> we please map out what needs to talk to what, and why?  Many of the use
> cases seem to be based on something that should be kicked off by the
> conductor, such as "migrate, resize, live-migrate" and it sounds like
> there are plans to make that happen.
>

i think the community is split on "anything can talk to anything". this 
was a side topic[1] on another thread a few months ago regarding 
versioned notifications(FUN!).

i agree with you that we should map what talks with what and why. i 
personally think it would reduce load and security risk on queue since 
the messages and the content would be tailored to consumer rather than 
the current grab bag dump. the counter argument was that it's a lot less 
flexible and contrary to the purpose of pub/sub.

[1] 
http://lists.openstack.org/pipermail/openstack-dev/2015-November/080215.html

cheers,

-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing

[Doug Hellmann]

> On Mar 21, 2016, at 9:43 PM, Adam Young  wrote:
> 
> I had a good discussion with the Nova folks in IRC today.
> 
> My goal was to understand what could talk to what, and the short according to 
> dansmith
> 
> " any node in nova land has to be able to talk to the queue for any other one 
> for the most part: compute->compute, compute->conductor, conductor->compute, 
> api->everything. There might be a few exceptions, but not worth it, IMHO, in 
> the current architecture."
> 
> Longer conversation is here:
> http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27
> 
> Right now, the message queue is a nightmare.  All sorts of sensitive 
> information flows over the message queue: Tokens (including admin) are the 
> most obvious.  Every piece of audit data. All notifications and all control 
> messages.
> 
> Before we continue down the path of "anything can talk to anything" can we 
> please map out what needs to talk to what, and why?  Many of the use cases 
> seem to be based on something that should be kicked off by the conductor, 
> such as "migrate, resize, live-migrate" and it sounds like there are plans to 
> make that happen.
> 
> So, let's assume we can get to the point where, if node 1 needs to talk to 
> node 2, it will do so only via the conductor.  With that in place, we can put 
> an access control rule in place:

Shouldn't we be trying to remove central bottlenecks by decentralizing 
communications where we can?

Doug


> 
> 1.  Compute nodes can only read from the queue 
> compute.-novacompute-.localdomain
> 2.  Compute nodes can only write to response queues in the RPC vhost
> 3.  Compute nodes can only write to notification queus in the notification 
> host.
> 
> I know that with AMQP, we should be able to identify the writer of a message. 
>  This means that each compute node should have its own user.  I have 
> identified how to do that for Rabbit and QPid.  I assume for 0mq is would 
> make sense to use ZAP (http://rfc.zeromq.org/spec:27) but I'd rather the 0mq 
> maintainers chime in here.
> 
> I think it is safe (and sane) to have the same use on the compute node 
> communicate with  Neutron, Nova, and Ceilometer.  This will avoid a false 
> sense of security: if one is compromised, they are all going to be 
> compromised.  Plan accordingly.
> 
> Beyond that, we should have message broker users for each of the components 
> that is a client of the broker.
> 
> Applications that run on top of the cloud, and that do not get presence on 
> the compute nodes, should have their own VHost.  I see Sahara on my Tripleo 
> deploy, but I assume there are others.  Either they completely get their own 
> vhost, or the apps should share one separate from the RPC/Notification vhosts 
> we currently have.  Even Heat might fall into this category.
> 
> Note that those application users can be allowed to read from the 
> notification queues if necessary.  They just should not be using the same 
> vhost for their own traffic.
> 
> Please tell me if/where I am blindingly wrong in my analysis.
> 
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing

[Adam Young]

I had a good discussion with the Nova folks in IRC today.

My goal was to understand what could talk to what, and the short 
according to dansmith


" any node in nova land has to be able to talk to the queue for any 
other one for the most part: compute->compute, compute->conductor, 
conductor->compute, api->everything. There might be a few exceptions, 
but not worth it, IMHO, in the current architecture."


Longer conversation is here:
 
http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27

Right now, the message queue is a nightmare.  All sorts of sensitive 
information flows over the message queue: Tokens (including admin) are 
the most obvious.  Every piece of audit data. All notifications and all 
control messages.


Before we continue down the path of "anything can talk to anything" can 
we please map out what needs to talk to what, and why?  Many of the use 
cases seem to be based on something that should be kicked off by the 
conductor, such as "migrate, resize, live-migrate" and it sounds like 
there are plans to make that happen.


So, let's assume we can get to the point where, if node 1 needs to talk 
to node 2, it will do so only via the conductor.  With that in place, we 
can put an access control rule in place:


1.  Compute nodes can only read from the queue 
compute.-novacompute-.localdomain

2.  Compute nodes can only write to response queues in the RPC vhost
3.  Compute nodes can only write to notification queus in the 
notification host.


I know that with AMQP, we should be able to identify the writer of a 
message.  This means that each compute node should have its own user.  I 
have identified how to do that for Rabbit and QPid.  I assume for 0mq is 
would make sense to use ZAP (http://rfc.zeromq.org/spec:27) but I'd 
rather the 0mq maintainers chime in here.


I think it is safe (and sane) to have the same use on the compute node 
communicate with  Neutron, Nova, and Ceilometer.  This will avoid a 
false sense of security: if one is compromised, they are all going to be 
compromised.  Plan accordingly.


Beyond that, we should have message broker users for each of the 
components that is a client of the broker.


Applications that run on top of the cloud, and that do not get presence 
on the compute nodes, should have their own VHost.  I see Sahara on my 
Tripleo deploy, but I assume there are others.  Either they completely 
get their own vhost, or the apps should share one separate from the 
RPC/Notification vhosts we currently have.  Even Heat might fall into 
this category.


Note that those application users can be allowed to read from the 
notification queues if necessary.  They just should not be using the 
same vhost for their own traffic.


Please tell me if/where I am blindingly wrong in my analysis.





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev