date:20140325

Re: [openstack-dev] [Nova][Neutron] Neutron + Nova + OVS security group fix

2014-03-25 Thread Akihiro Motoki

Hi Nachi and the teams,

(2014/03/26 9:57), Salvatore Orlando wrote:
> I hope we can sort this out on the mailing list IRC, without having to 
> schedule emergency meetings.
>
> Salvatore
>
> On 25 March 2014 22:58, Nachi Ueno mailto:na...@ntti3.com>> 
> wrote:
>
> Hi Nova, Neturon Team
>
> I would like to discuss issue of Neutron + Nova + OVS security group fix.
> We have a discussion in IRC today, but the issue is complicated so we 
> will have
> a conf call tomorrow 17:00 UST (10AM PDT). #openstack-neutron
>
> (I'll put conf call information in IRC)
>
>
> thanks, but I'd prefer you discuss the matter on IRC.
> I won't be available at that time and having IRC logs on eavesdrop will allow 
> me to catch up without having to ask people or waiting for minutes on the 
> mailing list.

I can't join the meeting too. It is midnight.

>
> <-- Please let me know if this time won't work with you.
>
> Bug Report
> https://bugs.launchpad.net/neutron/+bug/1297469
>
> Background of this issue:
> ML2 + OVSDriver + IptablesBasedFirewall combination is a default
> plugin setting in the Neutron.
> In this case, we need a special handing in VIF. Because OpenVSwitch
> don't support iptables, we are
> using linuxbride + openvswitch bridge. We are calling this as hybrid 
> driver.
>
>
> The hybrid solution in Neutron has been around for such a long time that I 
> would hardly call it a "special handling".
> To summarize, the VIF is plugged into a linux bridge, which has another leg 
> plugged in the OVS integration bridge.
>
> On the other discussion, we generalized the Nova  side VIF plugging to
> the Libvirt GenericVIFDriver.
> The idea is let neturon tell the VIF plugging configration details to
> the GenericDriver, and GerericDriver
> takes care of it.
>
>
> The downside of the generic driver is that so far it's assuming local 
> configuration values are sufficient to correctly determine VIF plugging.
> The generic VIF driver will use the hybrid driver if get_firewall_required is 
> true. And this will happen if the firewall driver is anything different from 
> the NoOp driver.
> This was uncovered by a devstack commit (1143f7e). When I previously 
> discussed with the people involved this issue, I was under the impression 
> that the devstack patch introduced the problem.
> Apparently the Generic VIF driver is not taking at the moments hints from 
> neutron regarding the driver to use, and therefore, from what I gather, makes 
> a decision based on nova conf flags only.
> So a quick fix would be to tell the Generic VIF driver to always use hybrid 
> plugging when neutron is enabled (which can be gathered by nova conf flags).
> This will fix the issue for ML2, but will either break or insert an 
> unnecessary extra hop for other plugins.

When the generic VIF driver is introduced, OVS VIF driver and the hybrid VIF 
driver are
considered same as e as both are pluggged into OVS and the hybrid driver is 
implemeted
as a variation of OVS driver, but the thing is not so simple than the first 
thought.
The hybrid driver solution lives such a long time and IMO the hybrid VIF driver 
should
be considered as a different one from OVS VIF driver. I start to think 
VIF_TYPE_OVS_HYBRID
is a good way as Savaltore mentioned below.

Another point to be discussed is whether passing vif secuirty attributes work 
from now on.
Even when neutron security group is enabled, do we need to do some port 
security mechanism
(anti-spoofing, )  on nova-compute side (such as libvirt nwfilter) or not?

>
>
> Unfortunatly, HybridDriver is removed before GenericDriver is ready
> for security group.
>
>
> The drivers were marked for deprecation in Havana, and if we thought the 
> GenericDriver was not good for neutron security groups we had enough time to 
> scream.
>
> This makes ML2 + OVSDriver + IptablesBasedFirewall combination 
> unfunctional.
> We were working on realfix, but we can't make it until Icehouse
> release due to design discussions [1].
>
> # Even if neturon side patch isn't merged yet.
>
> So we are proposing a workaround fix to Nova side.
> In this fix, we are adding special version of the GenericVIFDriver
> which can work with the combination.
> There is two point on this new Driver.
> (1) It prevent set conf.filtername. Because we should use
> NoopFirewallDriver, we need conf.filtername should be None
> when we use it.
> (2) use plug_ovs_hybrid and unplug_ovs_hybrid by enforcing
> get_require_firewall as True.

IIUC, the original intention of get_firewall_required() is to control
whether nwfilter is enabled or not, not to control hybird plugging.
As a plan, get_firewall_required() is changed to look binding:attribute
(binding:capablity:port_filter or binding:vif_security:iptable_required
if I use the concept discussed so far).
What we need is a way to determine hybrid plugging is required or not.

Re: [openstack-dev] [All][Keystone] Deprecation of the v2 API

2014-03-25 Thread Clint Byrum

Excerpts from Dolph Mathews's message of 2014-03-25 19:01:17 -0700:
> On Tue, Mar 25, 2014 at 5:50 PM, Russell Bryant  wrote:
> 
> > We discussed the deprecation of the v2 keystone API in the cross-project
> > meeting today [1].  This thread is to recap and bring that discussion to
> > some consensus.
> >
> > The issue is that Keystone has marked the v2 API as deprecated in Icehouse:
> >
> > https://blueprints.launchpad.net/keystone/+spec/deprecate-v2-api
> >
> > If you use the API, deployments will get this in their logs:
> >
> > WARNING keystone.openstack.common.versionutils [-] Deprecated: v2 API is
> > deprecated as of Icehouse in favor of v3 API and may be removed in K.
> >
> > The deprecation status is reflected in the API for end users, as well.
> > For example, from the CLI:
> >
> >   $ keystone discover
> >   Keystone found at http://172.16.12.38:5000/v2.0
> > - supports version v2.0 (deprecated) here
> > http://172.16.12.38:5000/v2.0/
> >
> > My proposal is that this deprecation be reverted.  Here's why:
> >
> > First, it seems there isn't a common use of "deprecated".  To me,
> > marking something deprecated means that the deprecated feature:
> >
> >  - has been completely replaced by something else
> >
> 
> >  - end users / deployers should take action to migrate to the
> >new thing immediately.
> >
> 
> >  - The project has provided a documented migration path
> 
> >  - the old thing will be removed at a specific date/release
> >
> 
> Agree on all points. Unfortunately, we have yet to succeed on the
> documentation front:
> 
> 
> https://blueprints.launchpad.net/keystone/+spec/document-v2-to-v3-transition
> 
> >
> > The problem with the above is that most OpenStack projects do not
> > support the v3 API yet.
> >
> > From talking to Dolph in the meeting, it sounds like the intention is:
> >
> >  - fully support v2, just don't add features
> >
> >  - signal to other projects that they should be migrating to v3
> >
> 
> Above all else, this was our primary goal: to raise awareness about our
> path forward, and to identify the non-obvious stakeholders that we needed
> to work with in order to drop support for v2. With today's discussion as
> evidence, I think we've succeeded in that regard :)
> 
> >
> > Given that intention, I believe the proper thing to do is to actually
> > leave the API marked as fully supported / stable.  Keystone should be
> > working with other OpenStack projects to migrate them to v3.  Once that
> > is complete, deprecation can be re-visited.
> >
> 
> Happy to!
> 
> Revert deprecation of the v2 API: https://review.openstack.org/#/c/82963/
> 
> Although I'd prefer to apply this patch directly to milestone-proposed, so
> we can continue into Juno with the deprecation in master.
> 

As somebody maintaining a few master-chasing CD clouds, I'd like to ask
you to please stop the squawking about deprecation until it has a definite
replacement and most if not all OpenStack core projects are using it.

1 out of every 2 API calls on these clouds produces one of these errors
in Keystone. That is just pointless. :-P

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] SR-IOV and IOMMU check

2014-03-25 Thread Gouzongmei

Hi, Yang, Yi y

Agree with you,  IOMMU and SR-IOV need to be checked beforehand.

I think it should be checked before booting a instance with the pci flavor, 
that means when the flavor contains some normal pci cards or SR-IOV cards. Just 
like when you find there are pci_requests in the instance system_metadata.

The details are out of my current knows.

Hope can help you.
From: Yang, Yi Y [mailto:yi.y.y...@intel.com]
Sent: Wednesday, March 26, 2014 10:51 AM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] SR-IOV and IOMMU check

Hi, all

Currently openstack can support SR-IOV device pass-through (at least there are 
some patches for this), but the prerequisite to this is both IOMMU and SR-IOV 
must be enabled correctly, it seems there is not a robust way to check this in 
openstack, I have implemented a way to do this and hope it can be committed 
into upstream, this can help find the issue beforehand, instead of letting kvm 
report the issue "no IOMMU found" until the VM is started. I didn't find an 
appropriate place to put into this, do you think this is necessary? Where can 
it be put into? Welcome your advice and thank you in advance.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] SR-IOV and IOMMU check

2014-03-25 Thread Yang, Yi Y

Hi, all

Currently openstack can support SR-IOV device pass-through (at least there are 
some patches for this), but the prerequisite to this is both IOMMU and SR-IOV 
must be enabled correctly, it seems there is not a robust way to check this in 
openstack, I have implemented a way to do this and hope it can be committed 
into upstream, this can help find the issue beforehand, instead of letting kvm 
report the issue "no IOMMU found" until the VM is started. I didn't find an 
appropriate place to put into this, do you think this is necessary? Where can 
it be put into? Welcome your advice and thank you in advance.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ironic][Keystone] Move drivers credentials to Keystone

2014-03-25 Thread Jay Faulkner


On 3/25/2014 1:50 PM, Matt Wagner wrote:

This would argue to me that the easiest thing for Ceilometer might be
to query us for IPMI stats, if the credential store is pluggable.
"Fetch these bare metal statistics" doesn't seem too off-course for
Ironic to me. The alternative is that Ceilometer and Ironic would both
have to be configured for the same pluggable credential store. 


There is already a blueprint with a proposed patch here for Ironic to do 
the querying: 
https://blueprints.launchpad.net/ironic/+spec/send-data-to-ceilometer.


I think, for terms of credential storage (and, for that matter, metrics 
gathering as I noted in that blueprint), it's very useful to have things 
pluggable. Ironic, in particular, has many different use cases: bare 
metal private cloud, bare metal public cloud, and triple-o. I could 
easily see all three being different enough to call for different forms 
of credential storage.


-Jay Faulkner

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][Neutron] Neutron + Nova + OVS security group fix

2014-03-25 Thread Nachi Ueno

Hi Salvatore


2014-03-25 17:57 GMT-07:00 Salvatore Orlando :
> I hope we can sort this out on the mailing list IRC, without having to
> schedule emergency meetings.

Russel requested to have a conf call on this, so let him decide it.

> Salvatore
>
> On 25 March 2014 22:58, Nachi Ueno  wrote:
>>
>> Hi Nova, Neturon Team
>>
>> I would like to discuss issue of Neutron + Nova + OVS security group fix.
>> We have a discussion in IRC today, but the issue is complicated so we will
>> have
>> a conf call tomorrow 17:00 UST (10AM PDT). #openstack-neutron
>>
>> (I'll put conf call information in IRC)
>
>
> thanks, but I'd prefer you discuss the matter on IRC.
> I won't be available at that time and having IRC logs on eavesdrop will
> allow me to catch up without having to ask people or waiting for minutes on
> the mailing list.
>
>>
>>
>> <-- Please let me know if this time won't work with you.
>>
>> Bug Report
>> https://bugs.launchpad.net/neutron/+bug/1297469
>>
>> Background of this issue:
>> ML2 + OVSDriver + IptablesBasedFirewall combination is a default
>> plugin setting in the Neutron.
>> In this case, we need a special handing in VIF. Because OpenVSwitch
>> don't support iptables, we are
>> using linuxbride + openvswitch bridge. We are calling this as hybrid
>> driver.
>>
>
> The hybrid solution in Neutron has been around for such a long time that I
> would hardly call it a "special handling".
> To summarize, the VIF is plugged into a linux bridge, which has another leg
> plugged in the OVS integration bridge.
>
>>
>> On the other discussion, we generalized the Nova  side VIF plugging to
>> the Libvirt GenericVIFDriver.
>> The idea is let neturon tell the VIF plugging configration details to
>> the GenericDriver, and GerericDriver
>> takes care of it.
>
>
> The downside of the generic driver is that so far it's assuming local
> configuration values are sufficient to correctly determine VIF plugging.
> The generic VIF driver will use the hybrid driver if get_firewall_required
> is true. And this will happen if the firewall driver is anything different
> from the NoOp driver.
> This was uncovered by a devstack commit (1143f7e). When I previously
> discussed with the people involved this issue, I was under the impression
> that the devstack patch introduced the problem. Apparently the Generic VIF
> driver is not taking at the moments hints from neutron regarding the driver
> to use, and therefore, from what I gather, makes a decision based on nova
> conf flags only.
> So a quick fix would be to tell the Generic VIF driver to always use hybrid
> plugging when neutron is enabled (which can be gathered by nova conf flags).
> This will fix the issue for ML2, but will either break or insert an
> unnecessary extra hop for other plugins.

get_firewall_required = True won't fix this issue. We need to make sure
we won't set config.filtername in this case.

>>
>> Unfortunatly, HybridDriver is removed before GenericDriver is ready
>> for security group.
>
>
> The drivers were marked for deprecation in Havana, and if we thought the
> GenericDriver was not good for neutron security groups we had enough time to
> scream.

The reason we missed this issue is we lack the negative test for
security groups..
so we couldn't realize this.

>> This makes ML2 + OVSDriver + IptablesBasedFirewall combination
>> unfunctional.
>> We were working on realfix, but we can't make it until Icehouse
>> release due to design discussions [1].
>>
>> # Even if neturon side patch isn't merged yet.
>>
>> So we are proposing a workaround fix to Nova side.
>> In this fix, we are adding special version of the GenericVIFDriver
>> which can work with the combination.
>> There is two point on this new Driver.
>> (1) It prevent set conf.filtername. Because we should use
>> NoopFirewallDriver, we need conf.filtername should be None
>> when we use it.
>> (2) use plug_ovs_hybrid and unplug_ovs_hybrid by enforcing
>> get_require_firewall as True.
>>
>> Here is patchs with UT.
>>
>> Workaournd fix:
>> Nova
>> https://review.openstack.org/#/c/82904/
>>
>> Devstack patch for ML2 (Tested with 82904)
>> https://review.openstack.org/#/c/82937/
>
>
> Are there other plugins which need the hybrid driver for sec groups to work?
> I think so.
> And also - the patch does not seem to work according to Jenkins. The
> failures look genuine to me.

I agere with you, however we should start with minimal fix for this.
We can work for another plugin in another patch.
This patch will fail in Jenkins because it needs 82904.

>>
>>
>> We have tested the patch 82904 with following test, and this works.
>>
>> - Launch VM
>> - Assign floating ip
>> - make sure ping to the floating ip is failing from GW
>> - modify security group rule to allow ping from anywhere
>> - make sure ping is working
>
>
> You can actually run your devstack patch with your patch under review in the
> check queue.
> Check what Aaron did here: https://review.openstack.org/#/c/78694/11

Nice hack. let me try it

Re: [openstack-dev] [All][Keystone] Deprecation of the v2 API

2014-03-25 Thread Dolph Mathews

On Tue, Mar 25, 2014 at 5:50 PM, Russell Bryant  wrote:

> We discussed the deprecation of the v2 keystone API in the cross-project
> meeting today [1].  This thread is to recap and bring that discussion to
> some consensus.
>
> The issue is that Keystone has marked the v2 API as deprecated in Icehouse:
>
> https://blueprints.launchpad.net/keystone/+spec/deprecate-v2-api
>
> If you use the API, deployments will get this in their logs:
>
> WARNING keystone.openstack.common.versionutils [-] Deprecated: v2 API is
> deprecated as of Icehouse in favor of v3 API and may be removed in K.
>
> The deprecation status is reflected in the API for end users, as well.
> For example, from the CLI:
>
>   $ keystone discover
>   Keystone found at http://172.16.12.38:5000/v2.0
> - supports version v2.0 (deprecated) here
> http://172.16.12.38:5000/v2.0/
>
> My proposal is that this deprecation be reverted.  Here's why:
>
> First, it seems there isn't a common use of "deprecated".  To me,
> marking something deprecated means that the deprecated feature:
>
>  - has been completely replaced by something else
>

>  - end users / deployers should take action to migrate to the
>new thing immediately.
>

>  - The project has provided a documented migration path


>  - the old thing will be removed at a specific date/release
>

Agree on all points. Unfortunately, we have yet to succeed on the
documentation front:


https://blueprints.launchpad.net/keystone/+spec/document-v2-to-v3-transition


>
> The problem with the above is that most OpenStack projects do not
> support the v3 API yet.
>
> From talking to Dolph in the meeting, it sounds like the intention is:
>
>  - fully support v2, just don't add features
>
>  - signal to other projects that they should be migrating to v3
>

Above all else, this was our primary goal: to raise awareness about our
path forward, and to identify the non-obvious stakeholders that we needed
to work with in order to drop support for v2. With today's discussion as
evidence, I think we've succeeded in that regard :)


>
> Given that intention, I believe the proper thing to do is to actually
> leave the API marked as fully supported / stable.  Keystone should be
> working with other OpenStack projects to migrate them to v3.  Once that
> is complete, deprecation can be re-visited.
>

Happy to!

Revert deprecation of the v2 API: https://review.openstack.org/#/c/82963/

Although I'd prefer to apply this patch directly to milestone-proposed, so
we can continue into Juno with the deprecation in master.


>
> In summary, until we have completed v3 support within OpenStack itself,
> it's premature to mark the API deprecated since that's a signal to end
> users and deployers that says action is required.
>
> Thoughts?
>
> [1]
>
> http://eavesdrop.openstack.org/meetings/project/2014/project.2014-03-25-21.01.log.html#l-103
>
> --
> Russell Bryant
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Murano][Heat] MuranoPL questions?

2014-03-25 Thread Zane Bitter

On 21/03/14 18:58, Stan Lagun wrote:

Zane,

I appreciate your explanations on Heat/HOT. This really makes sense.
I didn't mean to say that MuranoPL is better for Heat. Actually HOT is
good for Heat's mission. I completely acknowledge it.
I've tried to avoid comparison between languages and I'm sorry if it
felt that way. This is not productive as I don't offer you to replace

No problem, I didn't interpret it that way. I just wanted to use it as
an opportunity to document a bit more information about Heat works that
may be helpful to you.

HOT with MuranoPL (although I believe that certain elements of MuranoPL
syntax can be contributed to HOT and be valuable addition there). Also
people tend to protect what they have developed and invested into and to
be fair this is what we did in this thread to great extent.

Yes, we are all human and this is an unavoidable part of life :)

What I'm trying to achieve is that you and the rest of Heat team
understand why it was designed the way it is. I don't feel that Murano
can become full-fledged member of OpenStack ecosystem without a bless
from Heat team. And it would be even better if we agree on certain
design, join our efforts and contribute to each other for sake of
Orchestration program.

Thanks. To be clear, this is what I am looking for. At the beginning of
this thread I proposed a very simple possible implementation, and I'm
wanting folks to tell me what is missing from that model that would
justify a more complicated approach.

I'm sorry for long mail texts written in not-so-good English and
appreciate you patience reading and answering them.

Having said that let me step backward and explain our design decisions.

Cloud administrators are usually technical guys that are capable of
learning HOT and writing YAML templates. They know exact configuration
of their cloud (what services are available, what is the version of
OpenStack cloud is running) and generally understands how OpenStack
works. They also know about software they intent to install. If such guy
wants to install Drupal he knows exactly that he needs HOT template
describing Fedora VM with Apache + PHP + MySQL + Drupal itself. It is
not a problem for him to write such HOT template.

I'm aware that TOSCA has these types of constraints, and in fact I
suggested to the TOSCA TC that maybe this is where we should draw the
line between Heat and some TOSCA-compatible service: HOT should be a
concrete description of exactly what you're going to get, whereas some
other service (in this case Murano) would act as the constraints solver.
e.g. something like an image name would not be hardcoded in a Murano
template, you have some constraints about which operating system and
what versions should be allowed, and it would pick one and pass it to
Heat. So I am interested in this approach.

The worst outcome here would be to end up with something that was
equivalent to TOSCA but not easily translatable to the TOSCA Simple
Profile YAML format (currently a Working Draft). Where 'easily
translatable' preferably means 'by just changing some names'. I can't
comment on whether this is the case as things stand.

Note that such template would be designed for very particular
configuration. There are hundreds of combinations that may be used to
install that Drupal - use RHEL/Windows/etc instead of Fedora, use
ngnix/IIS/etc instead of Apache, use FastCGI instead of mod_php,
PostgreSQL instead of MySQL. You may choose to have all software on
single VM or have one VM for database and another for Drupal. There are
also constraints to those combinations. For example you cannot have
Fedora + IIS on the same VM. You cannot have Apache and Drupal on
different VMs.

So the HOT template represent fixed combination of those software
components. HOT may have input parameters like "username" or
"dbImageName" but the overall structure of template is fixed. You cannot
have template that choses whether to use Windows or Linux based on
parameter value.

As Thomas mentioned, CloudFormation now has conditionals. If we were to
add this feature in HOT (which seems likely), you would actually be able
to do that.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html

You cannot write HOT that accepts number of instances
it allowed to create and then decide what would be installed on each of
them. This is just not needed for Heat users.

The path we've gone down in Heat is to introduce software component
resources so that the definition of the software component to install is
separated from the definition of the server it runs on, and may even be
in a separate template file.

So you can write a template that launches a single server with (e.g.)
both Wordpress and MySQL installed, or you can have a separate template
(though with the conditional stuff above it could theoretically be the
same template even) that launches two servers with Wordpress on one and
MySQL on t

89 matches

Mail list logo