[openstack-dev] [neutron] Bug deputy report week of October 15th

[Brian Haley]

Hi,

I was Neutron bug deputy last week. Below is a short summary about 
reported bugs.


Note: I will not be at the team meeting this morning, sorry for the late 
notice.


-Brian


Critical bugs
-
None

High bugs
-

* https://bugs.launchpad.net/neutron/+bug/1798472 - Fullstack tests 
fails because process is not killed properly

  - gate failure

* https://bugs.launchpad.net/neutron/+bug/1798475 - Fullstack test 
test_ha_router_restart_agents_no_packet_lost failing

  - gate failure

* https://bugs.launchpad.net/neutron/+bug/1799124 - Path MTU discovery 
fails for VMs with Floating IP behind DVR routers

  - Needs confirmation, I took ownership

Medium bugs
---

* https://bugs.launchpad.net/neutron/+bug/1798577
  - Fix proposed, https://review.openstack.org/#/c/606007/

* A number of port-forwarding bugs were filed by Liu Yulong
  - https://bugs.launchpad.net/neutron/+bug/1799135
  - https://bugs.launchpad.net/neutron/+bug/1799137
  - https://bugs.launchpad.net/neutron/+bug/1799138
  - https://bugs.launchpad.net/neutron/+bug/1799140
  - https://bugs.launchpad.net/neutron/+bug/1799150
  - https://bugs.launchpad.net/neutron/+bug/1799155
  - Will discuss with Liu if he is working on them

Wishlist bugs
-

* https://bugs.launchpad.net/neutron/+bug/146 - When use ‘neutron 
net-update’, we cannot change the 'vlan-transparent' dynamically

  - not a bug as per the API definition, asked if proposing extension
  - perhaps possible to implement in backward-compatible way

* https://bugs.launchpad.net/neutron/+bug/1799178 - l2 pop doesn't 
always provide the whole list of fdb entries on agent restart

  - Need a smarter way to detect agent restarts

Invalid bugs


* https://bugs.launchpad.net/neutron/+bug/1798536 - OpenVswitch: qg-XXX 
goes to br-int instead of br-ext


* https://bugs.launchpad.net/neutron/+bug/1798689 -
Fullstack test test_create_one_default_qos_policy_per_project failed
  - Fixed by https://review.openstack.org/#/c/610280/

Further triage required
---

* https://bugs.launchpad.net/neutron/+bug/1798588 - 
neutron-openvswitch-agent break network connection on second reboot

  - Asked for more information from submitter

* https://bugs.launchpad.net/neutron/+bug/1798688 - iptables_hybrid 
tests 
tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_shelve_shelved_server 
failed

  - tempest.lib.exceptions.NotFound: Object not found
Details: {u'message': u'Instance None could not be found.', 
u'code': 404}

Not sure if issue with shelve/unshelve since the instance is gone

* https://bugs.launchpad.net/bugs/1798713 - [fwaas]wrong judgment in 
_is_supported_by_fw_l2_driver method

  - Fix proposed, https://review.openstack.org/#/c/605988
Need someone from FWaaS team to confirm and set priority

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][stable] Stable Core Team Update

[Brian Haley]

+1 from me :)

-Brian

On 10/02/2018 11:41 AM, Miguel Lavalle wrote:

Hi Stable Team,

I want to nominate Bernard Cafarrelli as a stable core reviewer for 
Neutron and related projects. Bernard has been increasing the number of 
stable reviews he is doing for the project [1]. Besides that, he is a 
stable maintainer downstream for his employer (Red Hat), so he can bring 
that valuable experience to the Neutron stable team.


Thanks and regards

Miguel

[1] 
https://review.openstack.org/#/q/(project:openstack/neutron+OR+openstack/networking-sfc+OR+project:openstack/networking-ovn)++branch:%255Estable/.*+reviewedby:%22Bernard+Cafarelli+%253Cbcafarel%2540redhat.com%253E%22 




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] subnet pool can not delete prefixes

[Brian Haley]

On 09/26/2018 10:11 PM, wenran xiao wrote:

Relation bug: https://bugs.launchpad.net/neutron/+bug/1792901
Any suggestion is welcome!


Removing a prefix from a subnetpool is not supported, there was an 
inadvertent change to the client that made it seem possible.  We are in 
the process of reverting it:


https://review.openstack.org/#/c/599633/

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Core status

[Brian Haley]

On 09/19/2018 02:19 PM, Gary Kotton wrote:

Hi,

I have recently transitioned to a new role where I will be working on 
other parts of OpenStack. Sadly I do not have the necessary cycles to 
maintain my core responsibilities in the neutron community. Nonetheless 
I will continue to be involved.


Thanks for all your work over the years, especially in keeping the 
reviews moving along on the neutron stable branches.  Good luck in your 
new role!


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Appointing Slawek Kaplonski to the Neutron Drivers team

[Brian Haley]

Congrats Slawek!

On 08/27/2018 12:42 PM, Miguel Lavalle wrote:

Dear Neutron team,

In order to help the Neutron Drivers team to perform its very important 
job of guiding the community to evolve the OpenStack Networking 
architecture to meet the needs of our current and future users [1], I 
have asked Slawek Kaplonski to join it. Over the past few years, he has 
gained very valuable experience with OpenStack Networking, both as a 
deployer  and more recently working with one of our key packagers. He 
played a paramount role in implementing our QoS (Quality of Service) 
features, currently leading that sub-team. He also leads the CI 
sub-team, making sure the prompt discovery and fixing of bugs in our 
software. On top of that, he is one of our most active reviewers, 
contributor of code to our reference implementation and fixer of bugs. I 
am very confident in Slawek making great contributions to the Neutron 
Drivers team.


Best regards

Miguel

[1] 
https://docs.openstack.org/neutron/latest/contributor/policies/neutron-teams.html#drivers-team



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Stable review

[Brian Haley]

On 07/12/2018 02:53 AM, Takashi Yamamoto wrote:

hi,

queens branch of networking-midonet has had no changes merged since
its creation.
the following commit would tell you how many gate blockers have been
accumulated.
https://review.openstack.org/#/c/572242/

it seems the stable team doesn't have a bandwidth to review subprojects
in a timely manner. i'm afraid that we need some policy changes.


In the future I would recommend just adding someone from the neutron 
stable team to the review, as we (I) don't have the bandwidth to go 
through the reviews of every sub-project.  Between Miguel, Armando, Gary 
and myself we can usually get to things pretty quickly. 
https://review.openstack.org/#/admin/groups/539,members


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] Bug deputy report

[Brian Haley]

Hi,

I was Neutron bug deputy last week. Below is a short summary about 
reported bugs.



Critical bugs
-
None

High bugs
-

* https://bugs.launchpad.net/neutron/+bug/1777908 -
Ensure _get_changed_synthetic_fields() return updatable fields - breaks 
consumers

  - Boden proposed reverts

* https://bugs.launchpad.net/neutron/+bug/1777922 - neutron is not 
dropping radvd privileges

  - Fix proposed, https://review.openstack.org/#/c/576923/

* https://bugs.launchpad.net/neutron/+bug/1777968 - Too many 
DBDeadlockError and IP address collision during port creating

  - Fix proposed, https://review.openstack.org/#/c/577739/

Medium bugs
---

* https://bugs.launchpad.net/neutron/+bug/1778118 - missing transaction 
in driver_controller.py for l3 flavors

  - Fix proposed, https://review.openstack.org/#/c/577246/

Wishlist bugs
-

* https://bugs.launchpad.net/neutron/+bug/146 - When use ‘neutron 
net-update’, we cannot change the 'vlan-transparent' dynamically

  - not a bug as per the API definition, asked if proposing extension
  - perhaps possible to implement in backward-compatible way

Further triage required
---

* https://bugs.launchpad.net/neutron/+bug/1777866 - QoS CLI – Warning in 
case when provided burst is lower than 80% BW

  - submitter wants CLI warning, not sure it's necessary as it is
already mentioned in the admin guide
  - possible change in OSC could address

* https://bugs.launchpad.net/neutron/+bug/1778407 - Quality of Service 
(QoS) in Neutron - Notes regarding burst (80% BW)

  - Closed as duplicate of 1777866

* https://bugs.launchpad.net/neutron/+bug/1777965 - Create port get 
quota related DBDeadLock


* https://bugs.launchpad.net/neutron/+bug/1778207 - fwaas v2 add port 
into firewall group failed

  - most likely need DVR/HA check in affected code path
  - need to ping SridarK to take a look

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [StarlingX] StarlingX code followup discussions

[Brian Haley]

On 05/23/2018 02:00 PM, Jeremy Stanley wrote:

On 2018-05-22 17:41:18 -0400 (-0400), Brian Haley wrote:
[...]

I read this the other way - the goal is to get all the forked code from
StarlingX into upstream repos.  That seems backwards from how this should
have been done (i.e. upstream first), and I don't see how a project would
prioritize that over other work.

[...]

I have yet to see anyone suggest it should be prioritized over other
work. I expect the extracted and proposed changes/specs
corresponding to the divergence would be viewed on their own merits
just like any other change and ignored, reviewed, rejected, et
cetera as appropriate.


Even doing that is work - going through changes, finding nuggets, 
proposing new specs I don't think we can expect a project to even go 
there, it has to be driven by someone already involved in StarlingX, IMHO.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [StarlingX] StarlingX code followup discussions

[Brian Haley]

On 05/22/2018 04:57 PM, Jay Pipes wrote:

Warning: strong opinions ahead.

On 05/22/2018 02:54 PM, Dean Troyer wrote:

Developers will need to re-create a repo locally in
order to work or test the code and create reviews (there are more git
challenges here). It would be challenging to do functional testing on
the rest of STX in CI without access to all of the code.


Please don't take this the wrong way, Dean, but you aren't seriously 
suggesting that anyone outside of Windriver/Intel would ever contribute 
to these repos are you?


What motivation would anyone outside of Windriver/Intel -- who must make 
money on this effort otherwise I have no idea why they are doing it -- 
have to commit any code at all to StarlingX?


I read this the other way - the goal is to get all the forked code from 
StarlingX into upstream repos.  That seems backwards from how this 
should have been done (i.e. upstream first), and I don't see how a 
project would prioritize that over other work.


I'm truly wondering why was this even open-sourced to begin with? I'm as 
big a supporter of open source as anyone, but I'm really struggling to 
comprehend the business, technical, or marketing decisions behind this 
action. Please help me understand. What am I missing?


I'm just as confused.

-Brian


My personal opinion is that I don't think that any products, derivatives 
or distributions should be hosted on openstack.org infrastructure.


Are any of the distributions of OpenStack listed at 
https://www.openstack.org/marketplace/distros/ hosted on openstack.org 
infrastructure? No. And I think that is completely appropriate.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] Bug deputy report

[Brian Haley]

Hi,

I was Neutron bug deputy last week. Below is a short summary about 
reported bugs.



Critical bugs
-
None

High bugs
-

* https://bugs.launchpad.net/bugs/1753504 - Remove mox/mox3 usage from 
testing

Multiple people have taken ownership of fixing this

* https://bugs.launchpad.net/bugs/1754062 - openstack client does not 
pass prefixlen when creating subnet

Looks like both a bug in the api-ref and in the openstacksdk

* https://bugs.launchpad.net/bugs/1754327 - Tempest scenario jobs 
failing due to no FIP connectivity

https://review.openstack.org/#/c/550832/ proposed to try and track
down reason for failure (thanks Slawek!)

* https://bugs.launchpad.net/bugs/1755243 - AttributeError when updating 
DvrEdgeRouter objects running on network nodes

Bug submitter has already posted a patch

* https://bugs.launchpad.net/bugs/1754563 - Arp_responder function has 
failed since Ocata

Looks like there is maybe a missing setup_privsep() call in the L2
agent code?  Yes, and Slawek just fixed it in master, so we'll need
to backport just the change in one file.  haleyb pushed a change.

Medium bugs
---

* https://bugs.launchpad.net/bugs/1753540 - When isolated/force metadata 
is enabled, metadata proxy doesn't get automatically started/stopped 
when needed

Daniel Alvarez sent patch for master and backports (thanks!)

* https://bugs.launchpad.net/bugs/1754770 - Duplicate iptables rule 
detected in Linuxbridge agent logs

Slawek took ownership as messages started after another change

Low bugs


* https://bugs.launchpad.net/bugs/1753384 - The old QoS policy ID is 
returned when updating the QoS policy ID, when the revision plugin is 
enabled

Guoshuai Li took ownership

Bugs that need further triage
-

* https://bugs.launchpad.net/bugs/1753434 - Unbound ports floating ip 
not working with address scopes in DVR HA

Found on Pike, need to determine if already fixed in master

* https://bugs.launchpad.net/bugs/1754695 - Incorrect state of the 
Openflow table

Restarting neutron_openvswitch_agent container fixed issue

* https://bugs.launchpad.net/bugs/1754600 - the detail for openstack 
quota show is not supported
Probably duplicate of 
https://bugs.launchpad.net/neutron/+bug/1716043 since that was opened to 
track CLI quota change


RFE bugs for drivers team
-

* https://bugs.launchpad.net/bugs/1753466 - [RFE] Support stateless 
security groups


* https://bugs.launchpad.net/bugs/1754123 - [RFE] Support filter with 
floating IP address substring



Thanks,
-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] [neutron] Current containerized neutron agents introduce a significant regression in the dataplane

[Brian Haley]

On 02/13/2018 05:08 PM, Armando M. wrote:



On 13 February 2018 at 14:02, Brent Eagles > wrote:


Hi,

The neutron agents are implemented in such a way that key
functionality is implemented in terms of haproxy, dnsmasq,
keepalived and radvd configuration. The agents manage instances of
these services but, by design, the parent is the top-most (pid 1).

On baremetal this has the advantage that, while control plane
changes cannot be made while the agents are not available, the
configuration at the time the agents were stopped will work (for
example, VMs that are restarted can request their IPs, etc). In
short, the dataplane is not affected by shutting down the agents.

In the TripleO containerized version of these agents, the supporting
processes (haproxy, dnsmasq, etc.) are run within the agent's
container so when the container is stopped, the supporting processes
are also stopped. That is, the behavior with the current containers
is significantly different than on baremetal and stopping/restarting
containers effectively breaks the dataplane. At the moment this is
being considered a blocker and unless we can find a resolution, we
may need to recommend running the L3, DHCP and metadata agents on
baremetal.


I didn't think the neutron metadata agent was affected but just the 
ovn-metadata agent?  Or is there a problem with the UNIX domain sockets 
the haproxy instances use to connect to it when the container is restarted?


There's quite a bit to unpack here: are you suggesting that running 
these services in HA configuration doesn't help either with the data 
plane being gone after a stop/restart? Ultimately this boils down to 
where the state is persisted, and while certain agents rely on 
namespaces and processes whose ephemeral nature is hard to persist, 
enough could be done to allow for a non-disruptive bumping of the afore 
mentioned services.


Armando - https://review.openstack.org/#/c/542858/ (if accepted) should 
help with dataplane downtime, as sharing the namespaces lets them 
persist, which eases what the agent has to configure on the restart of a 
container (think of what the l3-agent needs to create for 1000 routers).


But it doesn't address dnsmasq being unavailable when the dhcp-agent 
container is restarted like it is today.  Maybe one way around that is 
to run 2+ agents per network, but that still leaves a regression from 
how it works today.  Even with l3-ha I'm not sure things are perfect, 
might wind-up with two masters sometimes.


I've seen one suggestion of putting all these processes in their own 
container instead of the agent container so they continue to run, it 
just might be invasive to the neutron code.  Maybe there is another option?


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron][all] nova_metadata_ip removal

[Brian Haley]

Hi,

As part of the normal deprecate/removal process, the 'nova_metadata_ip' 
option is being removed from neutron as it was replaced with 
'nova_metadata_host' in the Pike cycle, 
https://review.openstack.org/#/c/518836/


Codesearch did find various repos still using the old value, so I posted 
a number of cleanups for them since it was pretty painless, 
https://review.openstack.org/#/q/topic:nova_metadata_ip_deprecated+(status:open+OR+status:merged)


This is just an FYI for anyone else that might trip over the old value 
going away once it finally merges, most will probably never notice.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Metering can't count traffic for floating ip, or internal ip.

[Brian Haley]

On 01/04/2018 09:50 PM, wenran xiao wrote:

hi all，
neutron metering can only count traffic that we send to 
*remote_ip*(egress), and *remote_ip* send to us(ingress), I think we 
should add method to count the traffic for floating ip or internal ip.

Any suggestions is welcome.


Neutron metering was originally created as a way to get input for 
billing tenants for usage, giving admins numbers for what stayed inside 
or exited a datacenter.  It has languished over the past few cycles 
either because it is working perfectly, or not used very much - my guess 
is the latter.


That said, if you want to propose an enhancement, there is an RFE 
process defined in 
https://docs.openstack.org/neutron/latest/contributor/policies/blueprints.html 
you can use, the neutron drivers team has weekly meetings to discuss 
things for inclusion into releases.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] performance issue between virtual networks

[Brian Haley]

On 12/27/2017 08:39 AM, Kim-Norman Sahm wrote:

Hi,

i've detected a performance issue by accessing an floating ip in a
different openstack network (same tenant).

example:
i have one tenant with two internal networks.
each network has its own vrouter which is connectet to the extnet.
the physical network infrastructure is 10Gbit/s.

          networkA
    VM1 --|   extnet
  ||vrouter1||
    VM2 --|  |
 |---ext
  networkB   |
    VM3 --|  |
  ||vrouter2||
    VM4 --|

VM1 -> VM2   ~8,6Gbit/s
VM3 -> VM4   ~8,6GBit/s
VM1 -> vrouter1  ~8.6GBit/s
VM4 -> vrouter2 ~8,6GBit/s
vrouter1 -> vrouter2 ~8,6Gbits
VM1 -> VM4   ~2,5GBit/s
VM1 -> vrouter2 ~2,5Gbit/s


I could only guess that vrouter1 and vrouter2 are on different nodes, so 
you're losing some performance going from virtual to physical and back 
(eg GSO).  Have you tried this for reference:


VM1 -> system on extnet
VM4 -> system on extnet

Also, are you sure when packets from VM1 -> VM4 leave the vrouter1 
interface they are still at the higher MTU?


-Brian




detected with iperf3
it's an openstack newton environment with openvswitch 2.6.1
VXLAN mtu is 8950 and 9000 for physical interfaces

does anybody has an idea what could be the cause of the performance
issue?

Best regards
Kim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Tis the season...for a cloud reboot

[Brian Haley]

On 12/19/2017 04:00 PM, Ben Nemec wrote:



On 12/19/2017 02:43 PM, Brian Haley wrote:

On 12/19/2017 11:53 AM, Ben Nemec wrote:

The reboot is done (mostly...see below).

On 12/18/2017 05:11 PM, Joe Talerico wrote:

Ben - Can you provide some links to the ovs port exhaustion issue for
some background?


I don't know if we ever had a bug opened, but there's some discussion 
of it in 
http://lists.openstack.org/pipermail/openstack-dev/2016-December/109182.html 
  I've also copied Derek since I believe he was the one who found it 
originally.


The gist is that after about 3 months of tripleo-ci running in this 
cloud we start to hit errors creating instances because of problems 
creating OVS ports on the compute nodes.  Sometimes we see a huge 
number of ports in general, other times we see a lot of ports that 
look like this:


Port "qvod2cade14-7c"
 tag: 4095
 Interface "qvod2cade14-7c"

Notably they all have a tag of 4095, which seems suspicious to me.  I 
don't know whether it's actually an issue though.


Tag 4095 is for "dead" OVS ports, it's an unused VLAN tag in the agent.

The 'qvo' here shows it's part of the VETH pair that os-vif created 
when it plugged in the VM (the other half is 'qvb'), and they're 
created so that iptables rules can be applied by neutron.  It's part 
of the "old" way to do security groups with the 
OVSHybridIptablesFirewallDriver, and can eventually go away once the 
OVSFirewallDriver can be used everywhere (requires newer OVS and agent).


I wonder if you can run the ovs_cleanup utility to clean some of these 
up?


As in neutron-ovs-cleanup?  Doesn't that wipe out everything, including 
any ports that are still in use?  Or is there a different tool I'm not 
aware of that can do more targeted cleanup?


Crap, I thought there was an option to just cleanup these dead devices, 
I should have read the code, it's either neutron ports (default) or all 
ports.  Maybe that should be an option.


-Brian

Oh, also worth noting that I don't think we have os-vif in this cloud 
because it's so old.  There's no os-vif package installed anyway.




-Brian

I've had some offline discussions about getting someone on this cloud 
to debug the problem.  Originally we decided not to pursue it since 
it's not hard to work around and we didn't want to disrupt the 
environment by trying to move to later OpenStack code (we're still 
back on Mitaka), but it was pointed out to me this time around that 
from a downstream perspective we have users on older code as well and 
it may be worth debugging to make sure they don't hit similar problems.


To that end, I've left one compute node un-rebooted for debugging 
purposes.  The downstream discussion is ongoing, but I'll update here 
if we find anything.




Thanks,
Joe

On Mon, Dec 18, 2017 at 10:43 AM, Ben Nemec  
wrote:

Hi,

It's that magical time again.  You know the one, when we reboot rh1 
to avoid

OVS port exhaustion. :-)

If all goes well you won't even notice that this is happening, but 
there is

the possibility that a few jobs will fail while the te-broker host is
rebooted so I wanted to let everyone know.  If you notice anything 
else
hosted in rh1 is down (tripleo.org, zuul-status, etc.) let me know. 
I have

been known to forget to restart services after the reboot.

I'll send a followup when I'm done.

-Ben

__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Tis the season...for a cloud reboot

[Brian Haley]

On 12/19/2017 11:53 AM, Ben Nemec wrote:

The reboot is done (mostly...see below).

On 12/18/2017 05:11 PM, Joe Talerico wrote:

Ben - Can you provide some links to the ovs port exhaustion issue for
some background?


I don't know if we ever had a bug opened, but there's some discussion of 
it in 
http://lists.openstack.org/pipermail/openstack-dev/2016-December/109182.html 
  I've also copied Derek since I believe he was the one who found it 
originally.


The gist is that after about 3 months of tripleo-ci running in this 
cloud we start to hit errors creating instances because of problems 
creating OVS ports on the compute nodes.  Sometimes we see a huge number 
of ports in general, other times we see a lot of ports that look like this:


Port "qvod2cade14-7c"
     tag: 4095
     Interface "qvod2cade14-7c"

Notably they all have a tag of 4095, which seems suspicious to me.  I 
don't know whether it's actually an issue though.


Tag 4095 is for "dead" OVS ports, it's an unused VLAN tag in the agent.

The 'qvo' here shows it's part of the VETH pair that os-vif created when 
it plugged in the VM (the other half is 'qvb'), and they're created so 
that iptables rules can be applied by neutron.  It's part of the "old" 
way to do security groups with the OVSHybridIptablesFirewallDriver, and 
can eventually go away once the OVSFirewallDriver can be used everywhere 
(requires newer OVS and agent).


I wonder if you can run the ovs_cleanup utility to clean some of these up?

-Brian

I've had some offline discussions about getting someone on this cloud to 
debug the problem.  Originally we decided not to pursue it since it's 
not hard to work around and we didn't want to disrupt the environment by 
trying to move to later OpenStack code (we're still back on Mitaka), but 
it was pointed out to me this time around that from a downstream 
perspective we have users on older code as well and it may be worth 
debugging to make sure they don't hit similar problems.


To that end, I've left one compute node un-rebooted for debugging 
purposes.  The downstream discussion is ongoing, but I'll update here if 
we find anything.




Thanks,
Joe

On Mon, Dec 18, 2017 at 10:43 AM, Ben Nemec  
wrote:

Hi,

It's that magical time again.  You know the one, when we reboot rh1 
to avoid

OVS port exhaustion. :-)

If all goes well you won't even notice that this is happening, but 
there is

the possibility that a few jobs will fail while the te-broker host is
rebooted so I wanted to let everyone know.  If you notice anything else
hosted in rh1 is down (tripleo.org, zuul-status, etc.) let me know.  
I have

been known to forget to restart services after the reboot.

I'll send a followup when I'm done.

-Ben

__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Stepping down from core

[Brian Haley]

Armando,

It is sad to see you step down, just wanted to thank you for your 
leadership and hard work, both upstream and downstream, it's definitely 
made neutron a better project for everyone.  Good luck wherever life 
takes you.


-Brian

On 12/15/2017 02:01 PM, Armando M. wrote:

Hi neutrinos,

To some of you this email may not come as a surprise.

During the past few months my upstream community engagements have been 
more and more sporadic. While I tried hard to stay committed and fulfill 
my core responsibilities I feel like I failed to retain the level of 
quality and consistency that I would have liked ever since I stepped 
down from being the Neutron PTL back at the end of Ocata.


I stated many times when talking to other core developers that being 
core is a duty rather than a privilege, and I personally feel like it's 
way overdue for me to recognize on the mailing list that it's the time 
that I state officially my intention to step down due to other commitments.


This does not mean that I will disappear tomorrow. I'll continue to be 
on neutron IRC channels, support the neutron team, being the release 
liasion for Queens, participate at meetings, and be open to providing 
feedback to anyone who thinks my opinion is still valuable, especially 
when dealing with the neutron quirks for which I might be (git) blamed :)


Cheers,
Armando



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Propose Slawek Kaplonski for Neutron core

[Brian Haley]

+1 from me!

On 11/29/2017 02:21 PM, Miguel Lavalle wrote:

Hi Neutron Team,

I want to nominate Slawek Kaplonski (irc: slaweq) to Neutron core. 
Slawek has been an active contributor to the project since the Mitaka 
cycle. He has been instrumental in the development of the QoS 
capabilities in Neutron, becoming the lead of the sub-team focused on 
that set of features. More recently, he has collaborated in the 
implementation of OVO and is an active participant in the CI sub-team. 
His number of code reviews during the Queens cycle is on par with the 
leading core members of the team: http://stackalytics.com/?module=neutron


In my opinion, his efforts are highly valuable to the team and we will 
be very lucky to have him as a fully voting core.


I will keep this nomination open for a week as customary,

Thank you,

Miguel


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Brian Haley]

On 10/30/2017 05:46 PM, Matthew Treinish wrote:
 From a quick glance at the logs my guess is that the issue is related 
to this stack trace in the l3 agent logs:


http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/neutron/neutron-l3-agent.txt.gz?level=TRACE#_2017-10-29_23_11_15_146

I'm not sure what's causing it to complain there. But, I'm on a plane 
right now (which is why this is a top post, sorry) so I can't really dig 
much more than that. I'll try to take a deeper look at things later when 
I'm on solid ground. (hopefully someone will beat me to it by then though)


I don't think that l3-agent trace is it, as the failure is coming from 
the API.  It's actually a trace that's happening due to the async nature 
of how the agent runs arping, fix is 
https://review.openstack.org/#/c/507914/ but it only removes the log noise.


http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/neutron/neutron-server.txt.gz 
has some tracebacks that look config related, possible missing DB table? 
 But I haven't looked very closely.


-Brian


On October 31, 2017 1:25:55 AM GMT+04:00, Mohammed Naser 
 wrote:


Hi everyone,

I'm looking for some help regarding an issue that we're having with
the Puppet OpenStack modules, we've had very inconsistent failures in
the Xenial with the following error:

 
http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/
 
http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/testr_results.html.gz
 Details: {u'message': u'Unable to associate floating IP
172.24.5.17   to fixed IP10.100.0.8  
 for instance
d265626a-77c1-4d2f-8260-46abe548293e. Error: Request to
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c
timed out', u'code': 400}

At this point, we're at a bit of a loss.  I've tried my best in order
to find the root cause however we have not been able to do this.  It
was persistent enough that we elected to go non-voting for our Xenial
gates, however, with no fix ahead of us, I feel like this is a waste
of resources and we need to either fix this or drop CI for Ubuntu.  We
don't deploy on Ubuntu and most of the developers working on the
project don't either at this point, so we need a bit of resources.

If you're a user of Puppet on Xenial, we need your help!  Without any
resources going to fix this, we'd unfortunately have to drop support
for Ubuntu because of the lack of resources to maintain it (or
assistance).  We (Puppet OpenStack team) would be more than happy to
work together to fix this so pop-in at #puppet-openstack or reply to
this email and let's get this issue fixed.

Thanks,
Mohammed



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Denver Team Dinner

[Brian Haley]

+1

On 09/12/2017 10:44 PM, Ihar Hrachyshka wrote:

+1

On Tue, Sep 12, 2017 at 9:44 PM, Kevin Benton  wrote:

+1

On Tue, Sep 12, 2017 at 8:50 PM, Sławek Kapłoński 
wrote:


+1

—
Best regards
Slawek Kaplonski
sla...@kaplonski.pl





Wiadomość napisana przez Miguel Lavalle  w dniu
12.09.2017, o godz. 17:23:

Dear Neutrinos,

Our social event will take place on Thursday September 12th at 7:30pm.
The venue is going to be https://www.famousdaves.com/Denver-Stapleton. It is
located 0.4 miles from the Renaissance Hotel, which translates to an easy 9
minutes walk.

I have a reservation for a group of 30 people under my name. Please
respond to this message with your attendance confirmation by Wednesday
night, so I can get a more accurate head count.

Looking forward to see y'all Thursday night

Best regards

Miguel

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] [ironic] [nova] Trying again on wait_for_compute in devstack

[Brian Haley]

On 08/02/2017 07:17 AM, Sean Dague wrote:

The 3 node scenarios in Neutron (which are still experimental nv) are
typically failing to bring online the 3rd compute. In cells v2 you have
to explicitly add nodes to the cells. There is a nova-manage command
"discover-hosts" that takes all the compute nodes which have checked in,
but aren't yet assigned to a cell, and puts them into a cell of your
choosing. We do this in devstack-gate in the gate.

However... subnodes don't take very long to setup (so few services). And
the nova-compute process takes about 30s before it's done all it's
initialization and actually checks in to the cluster. It's a real
possibility that discover_hosts will run before subnode 3 checks in. We
see it in logs. This also really could come and bite us on any multinode
job, and I'm a bit concerned some of the multinode jobs aren't running
multinode some times because of it.

One way to fix this, without putting more logic in devstack-gate, is
ensure that by the time stack.sh finishes, the compute node is up. This
was tried previously, but it turned out that we totally missed that it
broke Ironic (the check happened too early, ironic was not yet running,
so we always failed), Cells v1 (munges hostnames :(  ), and PowerVM
(their nova-compute was never starting correctly, and they were working
around it with a restart later).

This patch https://review.openstack.org/#/c/488381/ tries again. The
check is moved very late, Ironic seems to be running fine with it. Cells
v1 is just skipped, that's deprecated in Nova now, and we're not going
to use it in multinode scenarios. The PowerVM team fixed their
nova-compute start issues, so they should be good to go as well.


I had also filed https://bugs.launchpad.net/neutron/+bug/1707003 for 
this since it was mainly just affecting that one 3-node neutron job. 
Glad I hadn't started working on a patch, I'll just take a look at yours.


Thanks for working on it!

-Brian


This is an FYI that we're going to land this again soon. If you think
this impacts your CI / jobs, please speak up. The CI runs on both the
main and experimental queue on devstack for this change look pretty
good, so I think we're safe to move forward this time. But we also
thought that the last time, and were wrong.

-Sean




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] Proposal to change integrated neutron grenade gate job to multi-node

[Brian Haley]

Hi,

While looking at ways to reduce the number of jobs we run in the Neutron 
gate, I found we ran two very similar jobs for some projects:


gate-grenade-dsvm-neutron-ubuntu-xenial (single-node job)
gate-grenade-dsvm-neutron-multinode-ubuntu-xenial (2-node job)

We talked about this in the Neutron CI meeting this week [1] and felt it 
best to remove the single-node job and just run the multi-node job, 
mostly because it more mimics a "real" Neutron deployment where there 
are separate controller and compute nodes.  Looking at the Neutron 
grafana dashboard [2] the two jobs have about the same failure rate in 
the gate (~0), so I don't think there will be any problems with the switch.


This has an impact on the integrated gate since it currently runs the 
single-node job, so I wanted ot get thoughts on any issues they'd have 
with this change [3].


Thanks,

-Brian

[1] 
http://eavesdrop.openstack.org/meetings/neutron_ci/2017/neutron_ci.2017-07-11-16.00.log.html#l-112

[3] http://grafana.openstack.org/dashboard/db/neutron-failure-rate
[2] https://review.openstack.org/#/c/483600/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron][l3][dvr] Pike tasks and meeting consolidation

[Brian Haley]

At the Summit a few of us met (Miguel, Swami and myself) to talk about our
Pike tasks now that we have lost some resources.  We came up with this
short list, which can also be found on
https://etherpad.openstack.org/p/neutron-l3-subteam :

1. Support for configuring Floating IPs on Centralized router in DVR
environment
https://bugs.launchpad.net/neutron/+bug/1667877
2. Spec for Floating IP subnets in routed networks
3. DHCP Agent support for remote IP subnets (via a DHCP relay)
https://bugs.launchpad.net/neutron/+bug/1668145

We also decided that instead of holding two weekly meetings, one for L3 and
another for DVR, we'd just cover any DVR-related issues in the L3 meeting
(Thursdays 1500 UTC) since neither was taking a full hour.  Anyone is
welcome to add items to the agenda and attend to discuss them.

Thanks,

-Brian
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][all] So long, and thanks for all the fish

[Brian Haley]

John,

Thanks so much for all your work on a number of L3 items that wouldn't 
exist without you, IPv6 PD comes to mind :)  It's been great working 
with you, good luck!


-Brian

On 05/02/2017 06:07 AM, John Davidge wrote:

Friends and colleagues,

It is with a heavy heart that I write to say my adventure in the OpenStack
community is coming to an end. It began in 2012 with my first job as an
intern at Cisco, and ends here as the Technical Lead for Neutron in the
OpenStack Innovation Center at Rackspace.

In that time I’ve worked with a great many wonderful people from all
corners of this community, on a variety of projects that I’m proud to
include my name in the commit logs. Thank you all for creating an exciting
place to work, to debate, and occasionally to argue about the incredible
democratizing power that is OpenStack. Your passion and expertise are an
inspiration to the world.

Regretfully, I’m leaving a void in both the Neutron team and the OpenStack
Manuals team. Neutron will need a new Docs liaison, and OpenStack Manuals
will need a new lead for the Networking Guide. The cross-project work
we’ve done together over the last couple of cycles has been engaging and
fulfilling, and I encourage anyone interested in either or both roles to
get in touch with Kevin Benton and Alexandra Settle.

Good luck and best wishes to all of you in the future.

Until we meet again,

John



Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, Hyde 
Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at 
www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain 
confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. If 
you receive this transmission in error, please notify us immediately by e-mail at 
ab...@rackspace.com and delete the original message. Your cooperation is 
appreciated.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][networking-odl][qa] What's the recommended behavior for SG Rules with ethertype='IPv6' and protocol='icmp' ?

[Brian Haley]

On 04/03/2017 09:23 PM, Ghanshyam Mann wrote:

On Sun, Mar 26, 2017 at 9:31 PM, Sridhar Gaddam mailto:sgad...@redhat.com>> wrote:

On Fri, Mar 24, 2017 at 7:27 PM, Brian Haley mailto:haleyb@gmail.com>> wrote:

On 03/24/2017 06:41 AM, Ghanshyam Mann wrote:

Hi All,

Tempest is testing SG rule creation and pinging scenario
tests with
ethertype='IPv6' and protocol='icmp' [0].
In case of ethertype='IPv6', currently neutron accept
protocol type
as 'icmp', 'icmpv6' and 'ipv6-icmp' which again seems like
duplication
of SG rules bug on neutron side but not sure [1]

But it seems like some driver does not work with 'icmp' on
IPv6, at
least ODL as mentioned in bug [2]. Where few others like ML2/OVS
iptables driver convert 'icmp' to 'icmpv6' when
ethertype='IPv6' and had
no issue with 'icmp'.

IMO neutron should keep accepting 'icmp' for IPv6 for backward
compatibility and legacy usage and tempest should test
'icmp' also along
with other protocol type.
But we need more feedback on that what is right way (as per
backward
compatibility pov) and recommended way for having best
behaviour for SG
rules on IPv6. What best can work for all plugins also?


Thanks for raising this issue.  Let me just restate it a little
so it's clear.

1. One can create an IPv6 rule using protocol value "icmp"
today, and the base security group code does the right thing
changing the rule to be correct for the underlying
implementation, for example, "ipv6-icmp" for iptables.  It
doesn't look like all other drivers handle this properly.

​Well, let the Neutron API accept multiple values like
"icmp/icmpv6/ipv6-icmp", but IMO it should store a single Security
Group rule in the DB and raise "Duplicate error when similar rule is
configured once again".
Currently, Neutron treats each of them as a different Security Group
rule and stores them as separate entries in the DB.
However, IPtables Firewall driver in Neutron is converting[1] the
"ethertype=IPv6 and protocol=icmp" as a request to ICMPv6 and
applying the necessary ip-table rule.

https://github.com/openstack/neutron/blob/stable/newton/neutron/agent/linux/iptables_firewall.py#L373

<https://github.com/openstack/neutron/blob/stable/newton/neutron/agent/linux/iptables_firewall.py#L373>

Since this is not a documented behavior, other firewall drivers
(which I guess could be an issue even with OVS firewall driver) may
be missing this info.​

​++ for this, documentation could have helped this better way. ​

2. The neutron API will accept multiple values - "icmp",
"ipv6-icmp" and "icmpv6" for an IPv6 rule, but it will create
unique database entries for each (I just verified that).  While
that shouldn't create multiple entries in the base iptables
code, it will probably generate a warning in the logs about a
duplicate being suppressed.


So there are a few things that could be done:

1. Drivers need to accept "icmp" in order to be
backwards-compatible with the current code.

2. Duplicates should be detected and generate 409 (?) errors.

3. We should add a migration (IMO) where any duplicates are
squashed.

​Agree with your points.


I just created https://review.openstack.org/453346 as a start at #2, 
will try and add code to squash existing duplicates too (#3).


-Brian



​Yes, 409 on same type again make sense. And it can be documented
properly that 'icmp'​ will be treated same as other protocol type for
​
Ethertype=IPv6 and user will get 409 if they try to create and expect 2
different SG rules with those different protocol_type.
This is something change in current behavior where both requests('icmp'
and 'icmpv6') are treated as 2 separate rules. And so this new Tempest
tests will fail [1] which can be modified later.

With those points and current situation, I feel Tempest should tests the
current behavior proposed in [1] which will discover the drivers for
point 1. Later based on bug [2] consensus we can change the tests
accordingly.
I want to merge Tempest changes so that while changing anything on
neutron side, we know exactly what behavior we are going to change and
its impact etc.
Please leave comment on patch if any more feedback.



​

The open question is, do we want to change the DB to hav

Re: [openstack-dev] [neutron] - Adding Miguel Lavalle to neutron-drivers team

[Brian Haley]

+1 for Miguel!

On 03/30/2017 05:11 PM, Kevin Benton wrote:

Hi,

I would like to add Miguel Lavalle (mlavalle) to the Neutron drivers
team [1].

Miguel has been instrumental in implemented features across Neutron and
Nova (routed networks, DNS, etc) and is now leading the L3 team. He has
a very good high level architectural view of Neutron necessary for
deciding what features are feasible for the platform.



1. 
http://docs.openstack.org/developer/neutron/policies/neutron-teams.html#drivers-team

Cheers,
Kevin Benton


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] No DVR meeting today

[Brian Haley]

Both chairs have conflicts today, will resume the DVR meeting next week. 
 Current work items and bugs can be found on the meeting page as usual, 
https://wiki.openstack.org/wiki/Meetings/Neutron-DVR


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][networking-odl][qa] What's the recommended behavior for SG Rules with ethertype='IPv6' and protocol='icmp' ?

[Brian Haley]

On 03/24/2017 06:41 AM, Ghanshyam Mann wrote:

Hi All,

Tempest is testing SG rule creation and pinging scenario tests with
ethertype='IPv6' and protocol='icmp' [0].
In case of ethertype='IPv6', currently neutron accept protocol type
as 'icmp', 'icmpv6' and 'ipv6-icmp' which again seems like duplication
of SG rules bug on neutron side but not sure [1]

But it seems like some driver does not work with 'icmp' on IPv6, at
least ODL as mentioned in bug [2]. Where few others like ML2/OVS
iptables driver convert 'icmp' to 'icmpv6' when ethertype='IPv6' and had
no issue with 'icmp'.

IMO neutron should keep accepting 'icmp' for IPv6 for backward
compatibility and legacy usage and tempest should test 'icmp' also along
with other protocol type.
But we need more feedback on that what is right way (as per backward
compatibility pov) and recommended way for having best behaviour for SG
rules on IPv6. What best can work for all plugins also?


Thanks for raising this issue.  Let me just restate it a little so it's 
clear.


1. One can create an IPv6 rule using protocol value "icmp" today, and 
the base security group code does the right thing changing the rule to 
be correct for the underlying implementation, for example, "ipv6-icmp" 
for iptables.  It doesn't look like all other drivers handle this properly.


2. The neutron API will accept multiple values - "icmp", "ipv6-icmp" and 
"icmpv6" for an IPv6 rule, but it will create unique database entries 
for each (I just verified that).  While that shouldn't create multiple 
entries in the base iptables code, it will probably generate a warning 
in the logs about a duplicate being suppressed.



So there are a few things that could be done:

1. Drivers need to accept "icmp" in order to be backwards-compatible 
with the current code.


2. Duplicates should be detected and generate 409 (?) errors.

3. We should add a migration (IMO) where any duplicates are squashed.

The open question is, do we want to change the DB to have a different 
value here, like "icmpv6" ?  We could obviously add a migration where we 
update the value.  The problem is that flag day could pose a problem if 
out-of-tree drivers don't support the new value.  I think we should 
leave it "icmp" for that reason, thoughts from others?


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [goals] proposing a new goal: "Control Plane API endpoints deployment via WSGI"

[Brian Haley]

On 01/12/2017 08:40 PM, Emilien Macchi wrote:

Greetings OpenStack community,

I have been looking for a Community Goal [1] that would directly help
Operators and I found the "run API via WSGI" useful.
So I've decided to propose this one as a goal for Pike but I'll stay
open to postpone it to Queen if our community thinks we already have
too much goals for Pike.

Note that this goal might help to achieve 2 other goals later:
- enable and test SSL everywhere
- enable and test IPv6 everywhere


I know IPv6 is a secondary goal, but since this was previously working in 
devstack 
(https://www.openstack.org/summit/tokyo-2015/videos/presentation/deploying-and-operating-an-ipv6-only-openstack-cloud) 
I'll try and help out getting that working again 
(https://bugs.launchpad.net/devstack/+bug/1656329).  Added a line to the 
etherpad as well.  I know there's more to do than just that, but if we had a job 
in place to catch regressions at least we'd have a baseline that *something* works.


-Brian



Here's the draft:
https://review.openstack.org/419706

Any feedback is very welcome, thanks for reading so far.

[1] https://etherpad.openstack.org/p/community-goals




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] proposing Miguel Lavalle as neutron core and Brian Haley as L3 Lt

[Brian Haley]

+1 for Miguel, he's been doing a great job :)

On 12/15/2016 06:14 PM, Armando M. wrote:

Hi neutrinos,

Miguel Lavalle has been driving the project forward consistently and reliably. I
would like to propose him to be entrusted with +2/+A rights in the areas he's
been most prolific, which are L3 and DHCP.

At the same time, I'd like to propose Brian Haley as our next Chief L3 Officer.
Both of them have worked with Carl Baldwin extensively and that can only be a
guarantee of quality.

Cheers,
Armando

[1] https://review.openstack.org/#/c/411531/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] proposing Ryan Tidwell and Nate Johnston as service LTs

[Brian Haley]

+1

On 12/15/2016 06:58 PM, Armando M. wrote:

Hi neutrinos,

I would like to propose Ryan and Nate as the go-to fellows for service-related
patches.

Both are core in their repos of focus, namely neutron-dynamic-routing and
neutron-fwaas, and have a good understanding of the service framework, the agent
framework and other bits and pieces. At this point, entrusting them with the
responsibility is almost a formality.

Cheers,
Armando

[1] https://review.openstack.org/#/c/411536/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][DVR] - No IRC Meeting until 2017

[Brian Haley]

Hi Folks,

We will not have another DVR sub-team meeting until 2017, resuming on January 
4th, to accommodate those that will be away.


Enjoy the break!

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Proposing Abhishek Raut as neutronclient core

[Brian Haley]

+1

On 12/13/16 8:22 PM, Armando M. wrote:

Hi folks,

Abhishek Raut's recent involvement in OSC and python-neutronclient has
helped moving a few efforts along in the right direction. I would like
to suggest we double down on core firepower for the neutronclient repo
alongside Akihiro [1].

This not only will help speed up our transition to OSC CLI, but also
improve the number of folks who can effectively liasion with the OSC
team, and look after the needs of neutron's client repo.

Many thanks,
Armando

[1] https://review.openstack.org/#/c/410485/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][tricircle]DVR issue in cross Neutron networking

[Brian Haley]

On 12/05/2016 10:38 PM, joehuang wrote:

Hello, Brian,

Thank you for your comment, see inline comments marked with [joehuang].

The ASCII figure is not good in the plain text mail, you can check it at the 
browser: 
http://lists.openstack.org/pipermail/openstack-dev/2016-December/108447.html

Best Regards
Chaoyi Huang (joehuang)


From: Brian Haley [brian.ha...@hpe.com]
Sent: 06 December 2016 10:29
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron][tricircle]DVR issue in cross Neutron 
networking

On 12/5/16 3:03 AM, joehuang wrote:

Hello,


Hi Chaoyi,

Comments inline below.


 Tricircle plans to provide L2 network across Neutron to ease supporting
high
 availability of application:

 For example, in the following figure, the application is consisted of
 instance1 and instance2, these two instances will be deployed into two
 OpenStack. Intance1 will provide service through "ext net1"(i.e, external
 network in OpenStack1), and Instance2 will provide service through
 "ext net2". Instance1 and Instance2 will be plugged into same L2 network
 net3 for data replication( for example database replication ).

  +-+   +-+
  |OpenStack1   |   |OpenStack2   |
  | |   | |
  | ext net1|   | ext net2|
  |   +-+-+ |   |   +-+-+ |
  | |   |   | |   |
  | |   |   | |   |
  |  +--+--+|   |  +--+--+|
  |  | ||   |  | ||
  |  | R1  ||   |  | R2  ||
  |  | ||   |  | ||
  |  +--+--+|   |  +--+--+|
  | |   |   | |   |
  | |   |   | |   |
  | +---+-+-+   |   | +---+-+-+   |
  | net1  | |   | net2  | |
  |   | |   |   | |
  |  ++--+  |   |  ++--+  |
  |  | Instance1 |  |   |  | Instance2 |  |
  |  +---+  |   |  +---+  |
  | |   |   | |   |
  | |   | net3  | |   |
  |  +--+-++  |
  | |   | |
  +-+   +-+


Are Openstack1 and 2 simply different compute nodes?

[joehuang] No, OpenStack 1 and 2 are two OpenStack clouds, each OpenStack cloud
 includes its own services like Nova, Cinder, Neutron. That 
means
 R1, net1 are network objects in OpenStack cloud1 ( Neutron1)
 R2, net2 are network objects in OpenStack cloud2 ( Neutron2),
 but net3 will be a shared L2 network existing in both 
OpenStack cloud1 and
 OpenStack cloud2, i.e in Neutron 1 and Neutron 2.


So net3 is a shared private network, perhaps just a shared VLAN.  And something 
makes sure IP address allocations don't overlap.  Ok.



 When we deploy the application in such a way, no matter which part of the
 application stops providing service, the other part can still provide
 service, and take the workload from the failure one. It'll bring the
failure
 tolerance no matter the failure is due to OpenStack crush or upgrade, or
 part of the application crush or upgrade.

 This mode can work very well and helpful, and router R1 R2 can run in DVR
 or legacy mode.

 While during the discussion and review of the spec:
 https://review.openstack.org/#/c/396564/, in this deployment, the end user
 has to add two NICs for each instance, one for the net3(a L2 network across
 OpenStack). And the net3 (a L2 network across OpenStack) can not be allowed
 to add_router_interface to router R1 R2, this is not good in networking.

 If the end user wants to do so, there is DVR MAC issues if more than one L2
 network across OpenStack are performed add_router_interface to router
R1 R2.

 Let's look at the following deployment scenario:
 +-+   +---+
 |OpenStack1   |   |OpenStack2 |
 | |   |   |
 | ext net1|   | ext net2  |
 |   +-+-+ |   |   +-+-+   |
 | |   |   | | |
 | |   |   | | |
 | +---+--+|   |  +--+---+ |
 | |  ||   |  |  | |
 | |R1||   |  |   R2 | |
 | |  ||   |  |  | |
 | ++--+--+|   |  +--+-+-+ |
 |  |  |   |   | | |   |
 |  |  |   | net3  | | |   |
 |  |   -+-+---+-+--+  |   |
 |  || |   |   |   |   |
 |  | +--+---+ |   | +-+-+ |   |
 |  |

[openstack-dev] [Neutron][DVR] - No IRC Meeting this week

[Brian Haley]

Hi Folks,

We will not have a DVR sub-team meeting this week since neither Swami nor myself 
will be there to chair it.


We will resume our meetings next week on December 14th.

If you have any questions please ping us on IRC or send an email to the list.

https://wiki.openstack.org/wiki/Meetings/Neutron-DVR

Thanks,

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][tricircle]DVR issue in cross Neutron networking

[Brian Haley]

On 12/5/16 3:03 AM, joehuang wrote:

Hello,


Hi Chaoyi,

Comments inline below.


 Tricircle plans to provide L2 network across Neutron to ease supporting
high
 availability of application:

 For example, in the following figure, the application is consisted of
 instance1 and instance2, these two instances will be deployed into two
 OpenStack. Intance1 will provide service through "ext net1"(i.e, external
 network in OpenStack1), and Instance2 will provide service through
 "ext net2". Instance1 and Instance2 will be plugged into same L2 network
 net3 for data replication( for example database replication ).

  +-+   +-+
  |OpenStack1   |   |OpenStack2   |
  | |   | |
  | ext net1|   | ext net2|
  |   +-+-+ |   |   +-+-+ |
  | |   |   | |   |
  | |   |   | |   |
  |  +--+--+|   |  +--+--+|
  |  | ||   |  | ||
  |  | R1  ||   |  | R2  ||
  |  | ||   |  | ||
  |  +--+--+|   |  +--+--+|
  | |   |   | |   |
  | |   |   | |   |
  | +---+-+-+   |   | +---+-+-+   |
  | net1  | |   | net2  | |
  |   | |   |   | |
  |  ++--+  |   |  ++--+  |
  |  | Instance1 |  |   |  | Instance2 |  |
  |  +---+  |   |  +---+  |
  | |   |   | |   |
  | |   | net3  | |   |
  |  +--+-++  |
  | |   | |
  +-+   +-+


Are Openstack1 and 2 simply different compute nodes?


 When we deploy the application in such a way, no matter which part of the
 application stops providing service, the other part can still provide
 service, and take the workload from the failure one. It'll bring the
failure
 tolerance no matter the failure is due to OpenStack crush or upgrade, or
 part of the application crush or upgrade.

 This mode can work very well and helpful, and router R1 R2 can run in DVR
 or legacy mode.

 While during the discussion and review of the spec:
 https://review.openstack.org/#/c/396564/, in this deployment, the end user
 has to add two NICs for each instance, one for the net3(a L2 network across
 OpenStack). And the net3 (a L2 network across OpenStack) can not be allowed
 to add_router_interface to router R1 R2, this is not good in networking.

 If the end user wants to do so, there is DVR MAC issues if more than one L2
 network across OpenStack are performed add_router_interface to router
R1 R2.

 Let's look at the following deployment scenario:
 +-+   +---+
 |OpenStack1   |   |OpenStack2 |
 | |   |   |
 | ext net1|   | ext net2  |
 |   +-+-+ |   |   +-+-+   |
 | |   |   | | |
 | |   |   | | |
 | +---+--+|   |  +--+---+ |
 | |  ||   |  |  | |
 | |R1||   |  |   R2 | |
 | |  ||   |  |  | |
 | ++--+--+|   |  +--+-+-+ |
 |  |  |   |   | | |   |
 |  |  |   | net3  | | |   |
 |  |   -+-+---+-+--+  |   |
 |  || |   |   |   |   |
 |  | +--+---+ |   | +-+-+ |   |
 |  | | Instance1| |   | | Instance2 | |   |
 |  | +--+ |   | +---+ |   |
 |  |  | net4  |   |   |
 | ++---+--+---+-+ |
 |  |  |   |   |   |
 |  +---+---+  |   |  ++---+   |
 |  | Instance3 |  |   |  | Instance4  |   |
 |  +---+  |   |  ++   |
 | |   |   |
 +-+   +---+

 net3 and net4 are two L2 network across OpenStacks. These two networks will
 be added router interface to R1 R2. Tricircle can help this, and addressed
 the DHCP and gateway challenges: different gateway port for the same
network
 in different OpenStack, so there is no problem for north-south traffic, the
 north-south traffic will goes to local external network directly, for
example,
 Instance1->R1->ext net1, instance2->R2->ext net2.


Can you describe the subnet configuration here?  Is there just one per 
network and was is the IP range?



 The issue is in east-west traffic if R1 R2 are running in DVR mode:
 when instance1 tries to ping instance4, DVR MAC replacement will happen
before
 the packet leaves the host where the instance1 is running, when the packet
 arrives at the host where 

Re: [openstack-dev] [neutron][octavia] Neutron LBaaS governance change and Octavia to the big tent

[Brian Haley]

On 12/01/2016 08:54 AM, Ihar Hrachyshka wrote:

Armando M.  wrote:


Hi folks,

A few hours ago a governance change [1] has been approved by TC members. This
means that from now on, the efforts for Load Balancing as a Service efforts
rest officially in the hands of the Octavia PTL and the Octavia core team.

I will work with the help of the respective core teams to implement a smooth
transition. My suggestion at this point is for any ML communication that
pertain LBaaS issues to include [octavia] tag on the email subject.

Please do not hesitate to reach out for any questions and/or clarifications.

Cheers,
Armando

[1] https://review.openstack.org/#/c/313056/


Should we also move all neutron lbaas-tagged bugs to octavia in LP? And kill the
‘lbaas’ tag from doc/source/policies/bugs.rst?


Yes.  I added the lbaas bugs.rst tag removal to
https://review.openstack.org/#/c/404872/ already.  Can someone from Octavia 
close and/or move-over any remaining lbaas bugs?  I only see three at

https://bugs.launchpad.net/ubuntu/+source/neutron-lbaas

Thanks,

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Stepping down from core

[Brian Haley]

Hi Carl,

Thanks for all the hard work, Neutron is definitely better because of it.  Hope 
to work with you again some day.


Good luck with your new endeavor!

-Brian

On 11/17/2016 01:42 PM, Carl Baldwin wrote:

Neutron (and Openstack),

It is with regret that I report that my work situation has changed such that I'm
not able to keep up with my duties as a Neutron core reviewer, L3 lieutenant,
and drivers team member. My participation has dropped off considerably since
Newton was released and I think it is fair to step down and leave an opening for
others to fill. There is no shortage of talent in Neutron and Openstack and I
know I'm leaving it in good hands.

I will be more than happy to come back to full participation in Openstack and
Neutron in the future if things change again in that direction. This is a great
community and I've had a great time participating and learning with you all.

Well, I don't want to drag this out. I will still be around on IRC and will be
happy to help out where I am able. Feel free to ping me.

Carl


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] OVS agent reports error "ovs-ofctl: br-int is not a bridge or a socket"

[Brian Haley]

On 11/17/2016 03:14 AM, zhi wrote:

Hi, all

I build a devstack which code from master branch. By default the OVS version is
2.0.2 when install devstack. So I reinstall the OVS 2.6.0.

I meet an exception in OVS agent when all OVS services runs okay. Exception
shows below:

2016-11-17 15:47:29.175 ERROR neutron.agent.linux.utils
[req-2393c604-1938-4841-9ea9-8f691e6c2262 None None] Exit code: 1; Stdin:
hard_timeout=0,idle_timeout=0,priority=0,table=71,cookie=10671737546270118050,actions=drop;
Stdout: ; Stderr: ovs-ofctl: br-int is not a bridge or a socket


The error is saying br-int doesn't exist, and from your output below it doesn't. 
 Try running:


# ovs-vsctl add-br br-int

-Brian



I want to know why OVS agent tells me these info. All the OVS services run okay.
I use " ovs-vsctl show " to ensure OVS services and the output is right.

The detailed output when excute " ovs-vsctl show ":

root@devstack:~/openvswitch-2.6.0# ovs-vsctl show
6e695086-6663-4fd0-b713-5a12605c92cb
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-ex
Controller "tcp:127.0.0.1:6633 "
is_connected: true
fail_mode: secure
Port br-ex
Interface br-ex
type: internal
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
... ...

I think that OVS runs okay.

Did I miss some configuration or something about neutron or OVS?


Hope for your reply. :-)


Thanks
Zhi Chang


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][trunk-port] OVS tbr bridge wasn't be created by OVS agent

[Brian Haley]

On 11/15/16 5:12 AM, zhi wrote:

Sorry, I forgot to say my local environment is Liberty. :)


According to the blueprint and reviews this didn't land until Newton, 
maybe some in Mitaka, so I wouldn't expect it to work in Liberty.


-Brian



2016-11-15 18:07 GMT+08:00 zhi mailto:changzhi1...@gmail.com>>:

Hi, all

I followed this guide[1] to create trunk ports and created a vm
by using trunk port. But I met a weird problem. OVS agent didn't
generate " tbr " bridge. All the OVS bridges shows below:
"
[root@server-64 ~]# ovs-vsctl list-br
br-int
br-physnet4
br-tun
"
Why did the OVS agent doesn't create " tbr " bridge ? I think I must
miss something but I don't know.

I enabled " trunk " in service_plugins configuration in neutron
server. And I did not add anything in OVS agent. Did I miss any
configuration in OVS agent ?


Thanks
Zhi Chang

[1]: https://wiki.openstack.org/wiki/Neutron/TrunkPort#CLI_usage_example





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] HostNotCompatibleWithFixedIps exception happens when setting router's gateway.

[Brian Haley]

Hi Zhi,

Thanks for the report, comment below.

On 10/27/2016 05:04 AM, zhi wrote:

Hi, all.

I installed a devstack in my local environment. All the code from master
branch. After the installation, I have to show you some problems which I met.

First of all, I create an external network by this command " neutron
net-create public --router:external=True --provider:network_type=flat
--provider:physical_network=public ".

Secondly, I create a subnet with " subnet_type " by this command " neutron
subnet-create [net-id] 20.20.20.0/24  --service-types
list=true network:router_gateway ".

At last, I create a router and setting this router's gateway by this command
" neutron router-gateway-set [router-id] [net-id]".

Exception happens in Neutron Server, it says "
HostNotCompatibleWithFixedIps: Host devstack is not connected to a segment where
the existing fixed_ips on port 0f38ba01-8dd0-43de-92e3-b294bd4ebed8 will
function given the routed network topology. ".


Subnet service types is new in Newton, and it seems you've found a bug - can you 
file a bug on launchpad for it?


The one thing you might try to get past this is to disable DHCP on these 
subnets, but the error you linked seems different from [1].


-Brian

[1] https://bugs.launchpad.net/neutron/+bug/1636963


After I did some research about the exception,  I found this patch[1] was
adding this exception into neutron repo. I am confused about that. Why setting
router's gateway will trigger this exception? I don't execute any commands about
" routed_network ".

What's wrong ?

Could someone give some advice about that ? I upload all the network and
subnets info at here [2]. Detail exception at here [3].

BTW, what's the meaning of " tags " in network?

Hope for your reply. :)


Thanks
Zhi Chang


[1]. https://review.openstack.org/#/c/346217/3
[2]. http://paste.openstack.org/show/587157/
[3]. http://paste.openstack.org/show/587158/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] running non-devstack jobs in Python projects aka "it works outside devstack"

[Brian Haley]

On 10/14/2016 05:53 PM, Clark Boylan wrote:

On Thu, Oct 13, 2016, at 05:47 PM, Emilien Macchi wrote:

Greetings OpenStack,

== Background

Since the beginning of OpenStack (or almost), devstack has been used
as a common tool to deploy OpenStack in CI environment. Most of
OpenStack projects (if not all) that are written in Python use it to
deploy the different components.
While devstack became popular and the reference in term of deployment
tool for continuous integration, devstack doesn't deploy OpenStack in
production (versus some tools like Kolla, Fuel, TripleO, Juju, etc).
It means things might (and did) break when deploying OpenStack outside
devstack, for different reasons. Some examples:

* until recently, SSL was not tested, and I believe some projects
still don't test with SSL enabled.
* IPv6 is not tested everywhere.




IPv6 testing likely means two different things to two different groups
of people. First is whether or not the cloud endpoints are hosted over
IPv6 and the other is whether or not instances launched by openstack are
assigned IPv6 addresses. The second thing has been tested for quite a
while now (tempest has had tests for it for almost 2 years though it
hasn't been enabled in the gate for that long). We should definitely
ensure that we are testing with openstack servers listening on IPv6
addresses as well.


The first item - IPv6 service endpoints, is actually covered by an experimental 
job (tempest-dsvm-neutron-serviceipv6), and devstack supports it in local.conf:


SERVICE_IP_VERSION=6

Last year it was working great, bug I've noticed there are failures now, I'll 
take a crack at getting it all working again.  Maybe it's something we could 
then promote to voting?


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] floating IP is DOWN

[Brian Haley]

On 09/22/2016 10:19 AM, Barber, Ofer wrote:

when i assign a floating IP to a server, i see that the status of the floating
IP is "down"

why is that so ?

*_code:_*

LOG.info("\n<== float IP address: %s and status: %s  ==>" %
(float_ip['floating_ip_address'],float_ip['status']))

*_Output:_*

<== float IP address: 10.63.101.225 and status: DOWN  ==>


I couldn't find that code anywhere, what release was this on?

From a Newton-based system created yesterday, this is the message in the 
l3-agent log when I associate a floating IP:


Floating ip 4c1b4571-a003-43f2-96a1-f7073cd1319d added, status ACTIVE

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Adding ihrachys to the neutron-drivers team

[Brian Haley]

Congrats Ihar!

-Brian

On 09/17/2016 12:40 PM, Armando M. wrote:

Hi folks,

I would like to propose Ihar to become a member of the Neutron drivers team [1].

Ihar wide knowledge of the Neutron codebase, and his longstanding duties as
stable core, downstream package whisperer, release and oslo liaison (I am sure I
am forgetting some other capacity he is in) is going to put him at great comfort
in the newly appointed role, and help him grow and become wise even further.

Even though we have not been meeting regularly lately we will resume our
Thursday meetings soon [2], and having Ihar onboard by then will be highly
beneficial.

Please, join me in welcome Ihar to the team.

Cheers,
Armando

[1] 
http://docs.openstack.org/developer/neutron/policies/neutron-teams.html#drivers-team

[2] https://wiki.openstack.org/wiki/Meetings/NeutronDrivers



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra][neutron] IPv6 gate issue in OSIC cloud

[Brian Haley]

On 08/24/2016 04:52 PM, Brian Haley wrote:

Hi,

Starting sometime earlier in the week, gate jobs started failing that were
running in the OSIC cloud, due to a loss of connectivity to VMs running there
when Neutron L3 was configured.  I wanted to send some information out on what
we found in case other deployment tools trip over the same issue.


And I didn't mean to leave out all the people that helped track this down and 
expedite the patches - clarkb, dougwig, sc68cal, mordred, armax, sdague... many 
thanks everyone!


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [infra][neutron] IPv6 gate issue in OSIC cloud

[Brian Haley]

Hi,

Starting sometime earlier in the week, gate jobs started failing that were 
running in the OSIC cloud, due to a loss of connectivity to VMs running there 
when Neutron L3 was configured.  I wanted to send some information out on what 
we found in case other deployment tools trip over the same issue.


The problem was the VMs in OSIC are IPv6-only by default, so must be reachable 
using a global IPv6 address.  At boot, everything was fine - IPv6 address and 
default route were configured, but when IPv6 forwarding was enabled, poof!  The 
problem is that the default behavior in Linux is to remove any IPv6 default 
routes when forwarding is enabled, and that action caused connectivity loss to 
systems outside the OSIC datacenter using IPv6.  Luckily there's a sysctl to 
control this, and there is a patch out [1] that is working it's way through the 
check queue now.


So if you're creating puppet, chef, etc, scripts and deploying in an IPv6-only 
environment, you might need a few tweaks to not hit the same issue.


-Brian

[1] https://review.openstack.org/#/c/359490/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] devstack changing to neutron by default RSN

[Brian Haley]

On 08/05/2016 02:32 PM, Armando M. wrote:


>
> Looking at the health trend for DVR [1], the test hasn't failed in a
> while, so I wonder if this is induced by the proposed switch, even
> though I can't correlate it just yet (still waiting for caffeine to 
kick
> in). Perhaps we can give ourselves today to look into it and pull the
> trigger for 351450 > on Monday?
>
> [1]

http://status.openstack.org/openstack-health/#/job/gate-tempest-dsvm-neutron-dvr



The only functional difference in the new code that happens in the gate
is the iptables rule:

local default_dev=""
default_dev=$(ip route | grep ^default | awk '{print $5}')
sudo iptables -t nat -A POSTROUTING -o $default_dev -s
$FLOATING_RANGE -j MASQUERADE


I skipped this in [0], to give us further data pointsclasping at straws 
still.

[0] https://review.openstack.org/#/c/351876/


I haven't been able to reproduce it either, but it's unclear how packets would 
get into a VM on an island since there is no router interface, and the VM can't 
respond even if it did get it.


I do see outbound pings from the connected VM get to eth0, hit the masquerade 
rule, and continue on their way.  But those packets get dropped at my ISP since 
they're in the 10/8 range, so perhaps something in the datacenter where this is 
running is responding?  Grasping at straws is right until we see the results of 
Armando's test patch.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] devstack changing to neutron by default RSN

[Brian Haley]

On 08/05/2016 08:59 AM, Sean Dague wrote:

On 08/04/2016 09:15 PM, Armando M. wrote:

So glad we are finally within the grasp of this!

I posted [1], just to err on the side of caution and get the opportunity
to see how other gate jobs for Neutron might be affected by this change.

Are there any devstack-gate changes lined up too that we should be aware of?

Cheers,
Armando

[1] https://review.openstack.org/#/c/351450/


Nothing at this point. devstack-gate bypasses the service defaults in
devstack, so it doesn't impact that at all. Over time we'll want to make
neutron the default choice for all devstack-gate setups, and nova-net to
be the exception. But that actually can all be fully orthoginal to this
change.

The experimental results don't quite look in yet, it looks like one test
is failing on dvr (which is the one that tests for cross tenant
connectivity) -
http://logs.openstack.org/50/350750/5/experimental/gate-tempest-dsvm-neutron-dvr/4958140/

That test has been pretty twitchy during this patch series, and it's
quite complex, so figuring out exactly why it's impacted here is a bit
beyond me atm. I think we need to decide if that is going to get deeper
inspection, we live with the fails, or we disable the test for now so we
can move forward and get this out to everyone.


I took a quick look at this and can't reproduce it yet, here's what the test 
seems to do:


1a. Create a network/subnet (10.100.0.0/28)
 b. attach a router interface to the subnet
 c. boot VM1 on the network

2a. Create a network/subnet (10.100.0.16/28)
 b. do NOT attach a router interface to the subnet
 c. boot VM2 on the network

3. Ssh to VM1 and ping VM2 - it should fail since there's no route to the 
network, but it succeeds


The only place you should be able to ping that VM2 IP from is the dhcp 
namespace, which does work for me.


So if you are seeing it be flaky it could the VM placement (same host vs 
different host) is impacting it?  In the logs it showed the same hostId, but so 
did my test, so I don't have a good answer.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] stable/liberty busted

[Brian Haley]

On 08/04/2016 03:25 PM, Brian Haley wrote:

On 08/04/2016 03:16 PM, Rick Jones wrote:

On 08/04/2016 12:04 PM, Kevin Benton wrote:

Yeah, I wasn't thinking when I +2'ed that. There are two use cases for
the pinger, one for ensuring continuous connectivity and one for
eventual connectivity.

I think the revert is okay for a quick fix, but we really need a new
argument to the pinger for strictness to decide which behavior the test
is looking for.


What situation requires continuous connectivity?


Maybe the test names can answer that:

test_assert_pings_during_br_int_setup_not_lost()
_test_assert_pings_during_br_phys_setup_not_lost()

In other cases we want the previous behavior - is that IP alive?  It's probably
just best to put the old code back and make a new assert_async_ping() based on
this code.


https://review.openstack.org/351356

^^ that makes a new assert_async_ping() and restores assert_ping() to previous 
behavior.


It will need a few rechecks to verify it helps, although this error would be 
hard to trigger.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] stable/liberty busted

[Brian Haley]

On 08/04/2016 03:16 PM, Rick Jones wrote:

On 08/04/2016 12:04 PM, Kevin Benton wrote:

Yeah, I wasn't thinking when I +2'ed that. There are two use cases for
the pinger, one for ensuring continuous connectivity and one for
eventual connectivity.

I think the revert is okay for a quick fix, but we really need a new
argument to the pinger for strictness to decide which behavior the test
is looking for.


What situation requires continuous connectivity?


Maybe the test names can answer that:

test_assert_pings_during_br_int_setup_not_lost()
_test_assert_pings_during_br_phys_setup_not_lost()

In other cases we want the previous behavior - is that IP alive?  It's probably 
just best to put the old code back and make a new assert_async_ping() based on 
this code.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] stable/liberty busted

[Brian Haley]

On 08/04/2016 01:36 PM, Armando M. wrote:

Hi Neutrinos,

I have noticed that Liberty seems to be belly up [1]. I wonder if anyone knows
anything or has the time to look into it.

Many thanks,
Armando

[1] https://review.openstack.org/#/c/349039/


This could be due to this backport;

https://review.openstack.org/#/c/347062/

Before we were doing 'ping -c 3 -W 1 $IP', which will succeed as long as one 
packet is returned.


Now there is an outer loop that runs 'ping -c 1 -W 1 $IP', so a single dropped 
packet could cause an error.  Since sometimes that first packet causes ARP to 
happen, any delay near the 1-second mark looks like a lost packet, but is really 
just transient and packets 2 and 3 are fine.


I've started a revert and will recheck, but if I'm right an async issue like 
this is hard to reliably reproduce - I had to use iptables directly to test my 
theory about the return code from ping when 1/3 packets were lost.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][dvr][fip] fg device allocated private ip address

[Brian Haley]

On 08/02/2016 08:15 AM, huangdenghui wrote:

hi john and brain
   thanks for your information, if we get patch[1],patch[2] merged,then fg can
allocate private ip address. after that, we need consider floating ip dataplane,
in current dvr implementation, fg is used to reachment testing for floating ip,
now,with subnet types bp,fg has different subnet than floating ip address, from
fg'subnet gateway point view, to reach floating ip, it need a routes entry,
destination is some floating ip address, fg'ip address is the nexthop, and this
routes entry need be populated at the event of floating ip creating, deleting
when floating ip is dissociated. any comments?


Yes, there could be a small change necessary to the l3-agent in order to route 
packets with DVR enabled, but I don't see it being a blocker.


-Brian



On 2016-08-01 23:38 , John Davidge <mailto:john.davi...@rackspace.com> Wrote:

Yes, as Brian says this will be covered by the follow-up patch to [2]
which I¹m currently working on. Thanks for the question.

John


On 8/1/16, 3:17 PM, "Brian Haley"  wrote:

>On 07/31/2016 06:27 AM, huangdenghui wrote:
>> Hi
>>Now we have spec named subnet service types, which provides a
>>capability of
>> allowing different port of a network to allocate ip address from
>>different
>> subnet. In current implementation of DVR, fip also is distributed on
>>every
>> compute node, floating ip and fg's ip are both allocated from external
>>network's
>> subnets. In large public cloud deployment, current implementation will
>>consume
>> lots of public ip address. Do we need a RFE to apply subnet service
>>types spec
>> to resolve this problem. Any thoughts?
>
>Hi,
>
>This is going to be covered in the existing RFE for subnet service types
>[1].
>We currently have two reviews in progress for CRUD [2] and CLI [3], the
>IPAM
>changes are next.
>
>-Brian
>
>[1] https://review.openstack.org/#/c/300207/
>[2] https://review.openstack.org/#/c/337851/
>[3] https://review.openstack.org/#/c/342976/
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Rackspace Limited is a company registered in England & Wales (company
registered number 03897010) whose registered office is at 5 Millington Road,
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may
contain confidential or privileged information intended for the recipient.
Any dissemination, distribution or copying of the enclosed material is
prohibited. If you receive this transmission in error, please notify us
immediately by e-mail at ab...@rackspace.com and delete the original
message. Your cooperation is appreciated.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Belated nova newton midcycle recap (part 2)

[Brian Haley]

On 08/01/2016 10:15 PM, Matt Riedemann wrote:

Starting from where I accidentally left off:





We also talked a bit about live migration with Neutron. There has been a fix up
for live migration + DVR since Mitaka:

https://review.openstack.org/#/c/275073

It's a bit of a hacky workaround but the longer term solution that we all want (
https://review.openstack.org/#/c/309416 ) is not going to be in Newton and will
need discussion at the Ocata summit in Barcelona (John Garbutt was going to work
with the Neutron team on preparing for the summit on that one). So we agreed to
go with Swami's DVR fix but it needs to be rebased (which still hasn't happened
since the midcycle).


I just pushed an update to the DVR live-migration patch re-based to master, so 
feel free to review again.  Swami or myself will answer any other comments as 
quickly as possible.


Thanks,

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][dvr][fip] fg device allocated private ip address

[Brian Haley]

On 07/31/2016 06:27 AM, huangdenghui wrote:

Hi
   Now we have spec named subnet service types, which provides a capability of
allowing different port of a network to allocate ip address from different
subnet. In current implementation of DVR, fip also is distributed on every
compute node, floating ip and fg's ip are both allocated from external network's
subnets. In large public cloud deployment, current implementation will consume
lots of public ip address. Do we need a RFE to apply subnet service types spec
to resolve this problem. Any thoughts?


Hi,

This is going to be covered in the existing RFE for subnet service types [1]. 
We currently have two reviews in progress for CRUD [2] and CLI [3], the IPAM 
changes are next.


-Brian

[1] https://review.openstack.org/#/c/300207/
[2] https://review.openstack.org/#/c/337851/
[3] https://review.openstack.org/#/c/342976/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Proposing Jakub Libosvar for testing core

[Brian Haley]

+1

On 07/22/2016 04:12 AM, Oleg Bondarev wrote:

+1

On Fri, Jul 22, 2016 at 2:36 AM, Doug Wiegley mailto:doug...@parksidesoftware.com>> wrote:

+1


On Jul 21, 2016, at 5:13 PM, Kevin Benton mailto:ke...@benton.pub>> wrote:

+1

On Thu, Jul 21, 2016 at 2:41 PM, Carl Baldwin mailto:c...@ecbaldwin.net>> wrote:

+1 from me

On Thu, Jul 21, 2016 at 1:35 PM, Assaf Muller mailto:as...@redhat.com>> wrote:

As Neutron's so called testing lieutenant I would like to propose
Jakub Libosvar to be a core in the testing area.

Jakub has demonstrated his inherent interest in the testing area 
over
the last few years, his reviews are consistently insightful and his
numbers [1] are in line with others and I know will improve if given
the responsibilities of a core reviewer. Jakub is deeply involved 
with
the project's testing infrastructures and CI systems.

As a reminder the expectation from cores is found here [2], and
specifically for cores interesting in helping out shaping Neutron's
testing story:

* Guide community members to craft a testing strategy for features 
[3]
* Ensure Neutron's testing infrastructures are sufficiently
sophisticated to achieve the above.
* Provide leadership when determining testing Do's & Don'ts [4]. 
What
makes for an effective test?
* Ensure the gate stays consistently green

And more tactically we're looking at finishing the Tempest/Neutron
tests dedup [5] and to provide visual graphing for historical 
control
and data plane performance results similar to [6].

[1] http://stackalytics.com/report/contribution/neutron/90
[2]

http://docs.openstack.org/developer/neutron/policies/neutron-teams.html
[3]

http://docs.openstack.org/developer/neutron/devref/development.environment.html#testing-neutron
[4] https://assafmuller.com/2015/05/17/testing-lightning-talk/
[5] https://etherpad.openstack.org/p/neutron-tempest-defork
[6]

https://www.youtube.com/watch?v=a0qlsH1hoKs&feature=youtu.be&t=24m22s


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe


http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org
?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][DVR] - No IRC Meeting today

[Brian Haley]

Hi Folks,

Sorry for the late notice, but we will not have a DVR sub-team meeting today 
since neither Swami nor myself will be there to chair it.


We will resume our meetings next week on July 20th.

If you have any questions please ping us on IRC or send an email to the list.

https://wiki.openstack.org/wiki/Meetings/Neutron-DVR

Thanks,

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][DVR] - No IRC Meeting this week

[Brian Haley]

Hi Folks,

We will not have a DVR sub-team meeting this week since neither Swami nor myself 
will be there to chair it.


We will resume our meetings next week on July 6th.

If you have any questions please ping us on IRC or send an email to the list.

https://wiki.openstack.org/wiki/Meetings/Neutron-DVR

Thanks,

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Question about service subnets spec

[Brian Haley]

Thanks Carl for bringing this up, comments below.

On 05/26/2016 02:04 PM, Carl Baldwin wrote:

Hi folks,

Some (but not all) of you will remember a discussion we had about
service subnets at the last mid-cycle.  We've been iterating a little
bit on a spec [1] and we have just one issue that we'd like to get a
little bit more feedback on.

As a summary:  To me, the idea of this spec is to reserve certain
subnets for certain kinds of ports.  For example, DVR FIP gateway
ports, and router ports.  The goal of this is to be able to use
subnets with private addresses for these kinds of ports instead of
wasting public IP addresses.

The remaining question is how to expose this through the API.  I had
thought about just attaching a list of device_owners to the subnet
resource.  If a list is attached, then only ports with the right
device_owner will be allocated IP addresses from that subnet.  I
thought this would be an easy way to implement it and I thought since
device owner is already exposed through the API, maybe it would be
acceptable.  However, there is some concern that this exposes too much
of the internal implementation.  I understand this concern.

At the mid-cycle we had discussed some enumeration values that
combined several types to avoid having to allow a list of types on a
subnet.  They were going to look like this:

   dvr_gateway -> ["network:floatingip_agent_gateway"]
   router_gateway -> ["network:floatingip_agent_gateway",
"network:router_gateway"]

The idea was that we'd only allow one value for a subnet and the
difference between the two would be whether you wanted router ports to
use private IPs.  I think it would be clearer if we just have simpler
definitions of types and allow a list of them.


Yes, this was the original plan - two values (well, three since None was 
default), each mapping to a set of owners.



At this point the enumeration values map simply to device owners.  For example:

   router_ports -> "network:router_gateway"
   dvr_fip_ports -> "network:floatingip_agent_gateway"

It was at this point that I questioned the need for the abstraction at
all.  Hence the proposal to use the device owners directly.


I would agree, think having another name to refer to a device_owner makes it 
more confusing.  Using it directly let's us be flexible for deployers, and 
allows for using additional owners values if/when they are added.



Armando expressed some concern about using the device owner as a
security issue.  We have the following policy on device_owner:

   "not rule:network_device or rule:context_is_advsvc or
rule:admin_or_network_owner"

At the moment, I don't see this as much of an issue.  Do you?


I don't, since only admins should be able to set device_owner to these values 
(that's the policy we're talking about here, right?).


To be honest, I think Armando's other comment - "Do we want to expose 
device_owner via tha API or leave it an implementation detail?" is important as 
well.  Even though I think an admin should know this level of neutron detail, 
will they really?  It's hard to answer that question being so close to the code.


-Brian


[1] 
https://review.openstack.org/#/c/300207/3/specs/newton/subnet-service-types.rst




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][stable] proposing Brian Haley for neutron-stable-maint

[Brian Haley]

On 05/24/2016 11:17 AM, Ihar Hrachyshka wrote:



On 17 May 2016, at 13:07, Ihar Hrachyshka  wrote:

Hi stable-maint-core and all,

I would like to propose Brian for neutron specific stable team.

His stats for neutron stable branches are (last 120 days):

mitaka: 19 reviews; liberty: 68 reviews (3rd place in the top); kilo: 16 
reviews.

Brian helped the project with stabilizing liberty neutron/DVR jobs, and with 
other L3 related stable matters. In his stable reviews, he shows attention to 
details.

If Brian is added to the team, I will make sure he is aware of all stable 
policy intricacies.

Side note: recently I added another person to the team (Cedric Brandilly), and 
now I realize that I haven’t followed the usual approval process. That said, 
the person also has decent stable review stats, and is aware of the policy. If 
someone has doubts about that addition to the team, please ping me and we will 
discuss how to proceed.

Ihar


OK, it’s a whole week for the thread, and there are no objections. I added 
Brian to the neutron-stable-maint gerrit group.

Welcome Brian!


Thanks Ihar, and thanks to all for the +1's!

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][stable] proposing Brian Haley for neutron-stable-maint

[Brian Haley]

Ihar,

Thanks for the nomination, I'll just chime-in below.

On 05/19/2016 03:13 PM, Ihar Hrachyshka wrote:



On 19 May 2016, at 13:18, Tony Breeds  wrote:

On Thu, May 19, 2016 at 10:49:26AM +0200, Ihar Hrachyshka wrote:



On 19 May 2016, at 02:12, Tony Breeds  wrote:

On Tue, May 17, 2016 at 01:07:48PM +0200, Ihar Hrachyshka wrote:

Hi stable-maint-core and all,

I would like to propose Brian for neutron specific stable team.


+1

It'd be nice to see some comments on the reviews to indicate that the various
aspects of the policy have been thought about.

I know it gets a little repetitive but it's hard to assess the quality of
reviews without it :/


Meh. I am not sure we want to follow the rabbit into the deep hole.


Fair enough.  We can agree to disagree ;P


My problem with reviewing specific comments is that we never made it a 
requirement for nominations into other core teams. (When I was nominated to 
neutron-core, no one validated my specific comments; instead votes were based 
on general perception of my contributions.)

If we think it should be a requirement for stable maintainers, we may make it 
explicit in our stable policies, and then we may enforce it. [Not that I agree 
with such a requirement...]


I'll admit I typically haven't been judging stable backports from a policy 
perspective, I'm typically looking at them from the other end - now that we 
fixed this in master, is this broken in stable/foo and should it be backported? 
 And how far?  And does it make sense?  Basically, changing from a reactive to 
a proactive model.


I know most of the stable rules, and when I don't know what to do I usually just 
ask Ihar ;)



I say, people working with the person in question are in the best position to
judge. If you don’t pay attention to neutron branches (rightfully so!),
obviously it’s hard to assess.


Of course.  I agree with you.


That’s why we should have some sense of delegation in our stable team
hierarchy (stable-maint-core doing initial sanity checks, details left up to
project specific teams).


You say "should" here, which confuses me a little as I thought that's exactly
what we did.  Where do you feel like we should delegate, that we aren't?


We do delegate.

I only suggested that it’s hard for stable-maint-core to assess candidate 
involvement in project specific stable matters, hence we should generally trust 
choices made by project teams, as long as they pass basic sanity checks. 
Getting into too much detail, like checking candidate’s comments in gerrit, 
does seem like too much to me. Though basic validation that the person have a 
significant history of prior reviews does seem like a reasonable job for 
stable-maint-core though.

Stable releases are still supervised by stable PTL and liaisons, and are done 
in gerrit, where any issues with policy violations can be discussed, and where 
violators may be held responsible. If stable-maint-core will have problems with 
some candidates, we can always revoke permissions.


I'm part of the DVR sub-team in Neutron, so most of my focus is there, and it's 
been a pretty hot spot in the stable releases as people start to deploy it more. 
 I agreed to the nomination so I could be delegated in that little corner and 
expand from there, much as I did with upstream Neutron.  My goal is to increase 
the speed at which the downstream users (distros) can consume the stable 
branches and get fixes to their customers.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Timeframe for naming the P release?

[Brian Haley]

On 05/02/2016 02:53 PM, Shamail Tahir wrote:

Hi everyone,

When will we name the P release of OpenStack?  We named two releases
simultaneously (Newton and Ocata) during the Mitaka release cycle.  This gave us
the names for the N (Mitaka), N+1 (Newton), and N+2 (Ocata) releases.

If we were to vote for the name of the P release soon (since the location is now
known) we would be able to have names associated with the current release cycle
(Newton), N+1 (Ocata), and N+2 (P).  This would also allow us to get back to
only voting for one name per release cycle but consistently have names for N,
N+1, and N+2.


Is there really going to be an option besides Plymouth?  I remember something 
important happened there in 1620 ;-)


https://en.wikipedia.org/wiki/Plymouth,_Massachusetts

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Social at the summit

[Brian Haley]

+1

On 4/25/16 1:07 PM, Kyle Mestery wrote:

OK, there is enough interest, I'll find a place on 6th Street for us
and get a reservation for Thursday around 7 or so.

Thanks folks!

On Mon, Apr 25, 2016 at 12:30 PM, Zhou, Han  wrote:

+1 :)

Han Zhou
Irc: zhouhan


On Monday, April 25, 2016, Korzeniewski, Artur
 wrote:


Sign me up :)

Artur
IRC: korzen

-Original Message-
From: Darek Smigiel [mailto:smigiel.dari...@gmail.com]
Sent: Monday, April 25, 2016 7:19 PM
To: OpenStack Development Mailing List (not for usage questions)

Subject: Re: [openstack-dev] [neutron] Social at the summit

Count me in!
Will be good to meet all you guys!

Darek (dasm) Smigiel


On Apr 25, 2016, at 12:13 PM, Doug Wiegley
 wrote:



On Apr 25, 2016, at 12:01 PM, Ihar Hrachyshka 
wrote:

WAT???

It was never supposed to be core only. Everyone is welcome!


+2

irony intended.

Socials are not controlled by gerrit ACLs.  :-)

doug



Sent from my iPhone


On 25 Apr 2016, at 11:56, Edgar Magana 
wrote:

Would you extend it to ex-cores?

Edgar





On 4/25/16, 10:55 AM, "Kyle Mestery"  wrote:

Ihar, Henry and I were talking and we thought Thursday night makes
sense for a Neutron social in Austin. If others agree, reply on this thread
and we'll find a place.

Thanks!
Kyle

___
___ OpenStack Development Mailing List (not for usage
questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_
_ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
 OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Proposing Hirofumi Ichihara to Neutron Core Reviewer Team

[Brian Haley]

+1

On 4/8/16 6:58 AM, Henry Gessau wrote:

+1, Hirofumi will make a great addition.

Akihiro Motoki  wrote:

Hi Neutrinos,

As the API Lieutenant of Neutron team,
I would like to propose Hirofumi Ichihara (irc: hichihara) as a member of
Neutron core reviewer team mainly focuing on the API/DB area.

Hirofumi has been contributing neutron actively in the recent two
releases constantly.
He was involved in key features in API/DB areas in Mitaka such as
tagging support and network availability zones.
I believe his knowledge and involvement will be great addition to our team.
He have been reviewing constantly [1] and I expect he continue to work
for Newton or later.

Existing API/DB core reviews (and other Neutron core reviewers),
please vote +1/-1 for the addition of Hirofumi to the team.

Thanks!
Akihiro


[1] http://stackalytics.com/report/contribution/neutron/90



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Segments, subnet types, and IPAM

[Brian Haley]

On 03/28/2016 07:17 PM, Carl Baldwin wrote:

On Fri, Mar 11, 2016 at 10:04 PM, Salvatore Orlando
 wrote:

On 11 March 2016 at 23:15, Carl Baldwin  wrote:
I wonder if we could satisfy this requirement with tags - as it seems these
subnets are anyway operator-owned you should probably not worry about
regular tenants fiddling with them, and therefore the "helper" subnet needed
for the fip namespace could just be tagged to the purpose.


We discussed tags at the mid-cycle.  We decided against using them.
One reason is that tags are pretty new and are a moving target.
Another reason I was hesitant is that tags were designed to be user
facing.  As I recall, we didn't want the tags to affect code behavior.
But, maybe I misunderstood that.  We want service types to affect
IPAM, at least.

Brian Haley is going to put up a spec for this service type attribute
work.  Brian, would you mind posting a link to your spec review when
you've posted it?


The Subnet service-type attribute spec is at:

https://review.openstack.org/#/c/300207/

Be gentle :)

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Maintaining httplib2 python library

[Brian Haley]

On 03/17/2016 06:04 PM, Doug Wiegley wrote:

Here is the non comprehensive list of usages based on what trees I
happen to have checked out (which is quite a few, but not all of
OpenStack for sure).

I think before deciding to take over ownership of an upstream lib (which
is a large commitment over space and time), we should figure out the
migration cost. All the uses in Tempest come from usage in Glance IIRC
(and dealing with chunked encoding).

Neutron seems to use it for a couple of proxies, but that seems like
requests/urllib3 might be sufficient.


The Neutron team should talk to Cory Benfield (CC'd) and myself more about this 
if they run into problems. requests and urllib3 are a little limited with 
respect to proxies due to limitations in httplib itself.

Both of us might be able to dedicate time during the day to fix this if 
Neutron/OpenStack have specific requirements that requests is not currently 
capable of supporting.


Looks like neutron is using it to do HTTP requests via unix domain sockets. 
Unless I’m missing something, requests doesn’t support that directly. There are 
a couple of other libs that do, or we could monkey patch the socket. Or modify 
the agents to use localhost.


We have to use Unix domain sockets in the metadata proxy because it's running in 
a namespace, so can't use localhost to talk to the agent.  But we could use some 
other library of course.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat][Neutron] Mitaka RC1 available

[Brian Haley]

On 03/17/2016 09:13 PM, Armando M. wrote:



On 18 March 2016 at 00:16, Jeremy Stanley mailto:fu...@yuggoth.org>> wrote:

On 2016-03-17 09:44:59 +0530 (+0530), Armando M. wrote:
> Unfortunately, Neutron is also going to need an RC2 due to
> upstream CI issues triggered by infra change [1] that merged right
> about the same time RC1 was being cut.

Do you have any details on the impact that caused for Neutron? I
don't think I heard about it. Was there another ML thread I missed?


No, I didn't think a discussion was necessary. We did file bugs though:

https://bugs.launchpad.net/neutron/+bug/1558289
https://bugs.launchpad.net/neutron/+bug/1558397
https://bugs.launchpad.net/neutron/+bug/1558355


And the stable/liberty change for those still seeing netcat errors there:

https://review.openstack.org/#/c/294591/

I'll get it in stable/kilo too.

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][DVR] No Meeting tomorrow

[Brian Haley]

Sorry for the late notice, but I need to cancel the DVR meeting for 
tomorrow the 9th as a few of us have a conflict.  We'll resume again 
next week on the 16th.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] stack.sh keeps failing with RTNETLINK answers: Network is unreachable

[Brian Haley]

On 02/26/2016 03:48 AM, Paul Carlton wrote:

Sean

Don't think unstack failed, when I manually create the device ( sudo ovs-vsctl
--no-wait -- --may-exist add-br br-ex) unstack.sh removes it.

sudo ip route show
default via 172.18.20.1 dev eth0
169.254.169.254 via 172.18.20.2 dev eth0
172.18.20.0/24 dev eth0  proto kernel  scope link  src 172.18.20.23
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1


Matt

config attached


So getting back to your original error:

$ sudo ip route replace 10.1.0.0/20 via 192.168.100.200
RTNETLINK answers: Network is unreachable

Looking at your local.conf:

FIXED_RANGE=10.1.0.0/20
FLOATING_RANGE=192.168.100.0/24
Q_FLOATING_ALLOCATION_POOL=start=192.168.100.200,end=192.168.100.254
PUBLIC_NETWORK_GATEWAY=192.168.100.1
PUBLIC_BRIDGE=br-ex

The 'ip route replace...' above is adding a route for the fixed IP range you've 
specified for your private subnet via what should be the neutron router for the 
network.  That router was allocated .200 when it attached to the external 
subnet, which is fine.


In order for this to work, the IP $PUBLIC_NETWORK_GATEWAY (192.168.100.1) needs 
to have been configured on br-ex ($PUBLIC_BRIDGE) beforehand, since otherwise 
the network will be unreachable.  Does 'ip a s dev br-ex' show that IP?


It could be that some setting in your local.conf is causing some 
mis-configuration, pasting 'neutron port-list' and 'neutron subnet-list' might 
help track down what is wrong.


-Brian


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Intra-column wrapping in python-neutronclient

[Brian Haley]

On 2/24/16 3:58 PM, Steve Baker wrote:

On 25/02/16 06:23, Carl Baldwin wrote:

Hi,

I've noticed a new behavior from the python-neutronclient which
disturbs me.  For me, this just started happening with my latest build
of devstack which I built yesterday.  It didn't happen with another
recent but little bit older devstack.

The issue is that the client is now wrapping content within columns.
For example:


+-+-+--+

   | id  | name| subnets
   |

+-+-+--+

   | eb850219-6a42-42ed-ac6a-| public  |
099745e5-4925-4572-a88f- |
   | 927b03a0dc77| | a5376206c892
172.24.4.0/24   |
   | | | 5b6dfb0d-c97e-48ae-
   |
   | | | 98c9-7fe3e1e8e88b
2001:db8::/64  |
   | ec73110f-   | private | 4073b9e7-a58e-4d6e-
   |
   | 86ad-4292-9547-7c2789a7023b | | a2e4-7a45ae899671
10.0.0.0/24|
   | | |
f12aee80-fc13-4adf-a0eb- |
   | | | 706af4319094
fd9d:e27:3eab::/64  |

+-+-+--+


Notice how the ids in the first column are wrapped within the column.
I personally don't see this as an aesthetic improvement.  It destroys
by ability to cut and paste the data within this column.  When I
stretch my console out to fix it, I have to rerun the command with the
new window width to fix it.  I used to be able to stretch my console
horizontally and the wrapped would automatically go away.

My intention was that it be a usability improvement rather than merely
an aesthetic one. Yes, it is unfortunate that it affects this specific
copy paste scenario but there are others where it is improved. I've
often been in the situation where I don't know which uuid to copy
because of the amount of overlap of unrelated columns.


But even in your case, if the uuid of interest was wrapped it wouldn't 
be a candidate for cut/paste, so it's just as broken.



How can I turned this off now?  Also, can I request that this new
"feature" be disabled by default?


Table resizing only occurs when a tty is present. This means that any
existing script which parses table output will not be affected. It also
means that you can disable it by piping your command to cat.

If you're unwilling to adapt, or specify formatting options, or pipe to
cat, then I would recommend that you submit a change to cliff to read a
user set environment variable to switch off table resizing.


I don't think it's OK to change the formatting, then introduce a change 
to get the original behavior back, the change should be considered a 
bug.  If we want a different behavior then *that* should be controlled 
through some new option.


I filed https://bugs.launchpad.net/python-cliff/+bug/1549906

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][Neutron] IPv6 related intermittent test failures

[Brian Haley]

On 02/02/2016 10:03 PM, Matthew Treinish wrote:

On Tue, Feb 02, 2016 at 05:09:47PM -0800, Armando M. wrote:

Folks,

We have some IPv6 related bugs [1,2,3] that have been lingering for some
time now. They have been hurting the gate (e.g. [4] the most recent
offending failure) and since it looks like they have been without owners
nor a plan of action for some time, I made the hard decision of skipping
them [5] ahead of the busy times ahead.


So TBH I don't think the failure rate for these tests are really at a point
necessitating a skip:

http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_slaac
http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_dhcp6_stateless_from_os
http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_dhcp6_stateless_from_os

(also just a cool side-note, you can see the very obvious performance regression
caused by the keystonemiddleware release and when we excluded that version in
requirements)

Well, test_dualnet_dhcp6_stateless_from_os is kinda there with a ~10% failure
rate, but the other 2 really aren't. I normally would be -1 on the skip patch
because of that. We try to save the skips for cases where the bugs are really
severe and preventing productivity at a large scale.

But, in this case these ipv6 tests are kinda of out of place in tempest. Having
all the permutations of possible ip allocation configurations always seemed a
bit too heavy handed. These tests are also consistently in the top 10 slowest
for a run. We really should have trimmed down this set a while ago so we're only
have a single case in tempest. Neutron should own the other possible
configurations as an in-tree test.

Brian Haley has a patch up from Dec. that was trying to clean it up:

https://review.openstack.org/#/c/239868/


I just updated that to mark six of the eight tests as "slow" per your previous 
comment, such that only the dual-NIC/dual-stack tests are run in the gate, the 
others will run in the periodic nightly job.


http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-all-master

Will help lessen the impact until we can determine if it's the test or Neutron.

-Brian


We probably should revisit that soon, since quite clearly no one is looking at
these right now.


-Matt Treinish




Now one might argue that skipping them is counterproductive because it may
allow other regressions to sneak in, but I am hoping that this
controversial action will indeed smoke out the right folks.

Comments welcome.

Regards,
Armando

[1] https://bugs.launchpad.net/neutron/+bug/1477192
[2] https://bugs.launchpad.net/neutron/+bug/1509004
[3] https://bugs.launchpad.net/openstack-gate/+bug/1540983
[4]
http://logs.openstack.org/37/264937/5/gate/gate-tempest-dsvm-neutron-full/afeaabd//logs/testr_results.html.gz
[5] https://review.openstack.org/#/c/275457/




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] gate-neutron-dsvm-api failures

[Brian Haley]

On 01/14/2016 05:42 PM, John Davidge (jodavidg) wrote:

The 
neutron.tests.api.admin.test_floating_ips_admin_actions.FloatingIPAdminTestJSON
test has been consistently failing for patch
https://review.openstack.org/#/c/258754/ and I don’t see how they can be
related. This patch has been trying to merge for a month.

This test seems to be experiencing a lot of failures recently:

http://status.openstack.org//elastic-recheck/data/uncategorized.html#gate-neutron-dsvm-api

Has it been diagnosed? Could somebody more familiar with the test take a look
please?


John,

That test was just recently changed too:

https://review.openstack.org/#/c/265016/2/neutron/tests/api/admin/test_floating_ips_admin_actions.py

So perhaps that change didn't actually fix things.

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] Testing Neutron with latest OVS

[Brian Haley]

On 1/14/16 9:28 AM, Russell Bryant wrote:


The consensus I'm picking up roughly is that for those working on the
features, testing with source builds seems to be working fine.  It's
just not something anyone wants to gate the main Neutron repo with.
That seems quite reasonable.  If the features aren't in proper
releases yet, I don't see gating as that important anyway.


Yes, I don't see it as being used for gating (yet), just for easily 
selecting a version of OVS I want to use for testing, like something 
that supports IPv6 tunnels, whether it's pre-built or built by the script.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Infra] Gerrit Upgrade to ver 2.11, completed.

[Brian Haley]

On 2015-12-16 16:24, openstack-dev-requ...@lists.openstack.org wrote:

Thanks to everyone for their patience while we upgraded to Gerrit
2.11.  I'm happy to announce that we were able to successfully
completed this task at around 21:00 UTC.  You may hack away once more.

If you encounter any problems, please let us know here or in
#openstack-infra on Freenode.


I'm still undecided on 2.11, have to give it more time, but I have noticed one 
thing that's annoying...


Trying to copy text from a review no longer works easily.  When I highlight text 
there's a little "bubble" pop-up of {press c to comment}, which seems to 
interfere with both my three-button mouse copy buffer, as well as Ctrl-C.  Call 
me a nitpicker, but having to highlight text, right-button click, Copy, 
right-button click, Paste, is a pain.


Maybe someone has a simple work-around for that.

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack]host not reachable with iptables reject after init

[Brian Haley]

On 11/09/2015 09:55 AM, Wilence Yao wrote:

Hi all,
After I run devstack/stack.sh completely, I found that api is not reachable.
After some check, I found some iptables rules cause the problem:





ACCEPT tcp  -- 0.0.0.0/0  0.0.0.0/0 
state NEW tcp dpt:22
REJECT all  -- 0.0.0.0/0  0.0.0.0/0 
reject-with icmp-host-prohibited
```

The last  two rules reject all access to the host except port 22(ssh). Why
should devstack add this two rules in host?


The devstack scripts don't add either of those rules, my guess is your distro 
has locked things down by default.  So you'll need to figure out how best to 
deal with it, either disabling completely or opening all the ports you'll need 
for devstack to function.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] Weekly DVR Meeting starting next week

[Brian Haley]

A few of us had a discussion this week at Summit and decided to re-start 
the weekly Neutron Distributed Virtual Router (DVR) meeting.  The goal 
is to help:


- Stabilize DVR - fix the bugs
- Address performance/scalability issues
- Get the DVR jobs voting again

Meetings will be on Wednesdays starting next week at 15:00 UTC.  I'm in 
the process of updating 
http://eavesdrop.openstack.org/#Neutron_Distributed_Virtual_Router_Meeting 
with a link to the meeting page and agenda, which is currently at 
https://wiki.openstack.org/wiki/Meetings/Neutron-DVR


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] PTL Non-Candidacy

[Brian Haley]

On 09/12/2015 01:14 PM, Armando M. wrote:


On 12 September 2015 at 18:38, Gary Kotton mailto:gkot...@vmware.com>> wrote:

Thanks! You did a great job. Looking back you made some very tough and
healthy decisions. Neutron has a new lease on life!
It is tradition that the exiting PTL buy drinks for the community :)

Ok, none of these kind words make you change your mind? This project needs you!


I can only add kind words myself, thanks for all the hard work over the past 
three cycles Kyle!


-Brian



From: "mest...@mestery.com "
mailto:mest...@mestery.com>>
Reply-To: OpenStack List mailto:openstack-dev@lists.openstack.org>>
Date: Saturday, September 12, 2015 at 12:12 AM
To: OpenStack List mailto:openstack-dev@lists.openstack.org>>
Subject: [openstack-dev] [neutron] PTL Non-Candidacy

I'm writing to let everyone know that I do not plan to run for Neutron PTL
for a fourth cycle. Being a PTL is a rewarding but difficult job, as Morgan
recently put it in his non-candidacy email [1]. But it goes further than
that for me. As Flavio put it in his post about "Being a PTL" [2], it's a
full time job. In the case of Neutron, it's more than a full time job, it's
literally an always on job.

I've tried really hard over my three cycles as PTL to build a stronger web
of trust so the project can grow, and I feel that's been accomplished. We
have a strong bench of future PTLs and leaders ready to go, I'm excited to
watch them lead and help them in anyway I can.

As was said by Zane in a recent email [3], while Heat may have pioneered the
concept of rotating PTL duties with each cycle, I'd like to highly encourage
Neutron and other projects to do the same. Having a deep bench of leaders
supporting each other is important for the future of all projects.

See you all in Tokyo!
Kyle

[1]

http://lists.openstack.org/pipermail/openstack-dev/2015-September/074157.html
[1]

http://lists.openstack.org/pipermail/openstack-dev/2015-September/073986.html
[2]

http://lists.openstack.org/pipermail/openstack-dev/2015-September/074242.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][L3] [API]: API to get the list of router ports

[Brian Haley]

On 08/13/2015 04:04 AM, Padmanabhan Krishnan wrote:

Hello,
Is there a Neutron public API to get the list of router ports? Something similar
to what the command "neutron router-port-list {tenant}" gives.


router-port-list takes a router as an argument, not a tenant.


I wasn't able to find one in the Neutron API doc as well as in
"neutronclient/v2_0/client.py".

I think with a combination of subnet_show, port_list, one can find the list of
neutron router ports, but just wanted to see if there's an API already 
available.


$ neutron port-list -- --device_owner=network:router_interface

or 'router_interface_distributed' if you have DVR enabled.

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] What are problems of Distributed SNAT?

[Brian Haley]

On 07/24/2015 08:17 AM, Miyagishi, Takanori wrote:

Dear Carl,

Thank you for your information!

I checked the etherpad, and I propose a new idea of Distributed SNAT 
implementation.
Following figure is my idea, "Shared SNAT Address per Tenant per Compute Node".


I think this is the "One IP Address per Router per Host" item in the etherpad 
since each distributed router will consume one IP.



   +---+---+--+
   |   |eth|   TenantA : TenantB  |
   |   +-+-+   :  external-network|
   | | :  10.0.0.0/24 |
   | +--++--- |
   ||  : ||
   ||10.0.0.100: |10.0.0.101  |
   |+---+ +-+-+ +---+  :  +--+---+ +---+  |  ++
   || R | |   SNAT| | R |  :  | SNAT | | R |  |  ||
   |+-+-+ +-+---+-+ +-+-+  :  +--+---+ +-+-+  |  ... ||
   |  | |   | |: |   ||  ||
   |  | |   | |: |   ||  ++
   |   ---+--+--+--   --+--+--+--- :  ---+---+--- |  Compute Node N
   | | |   : ||
   |  +--+--+   +--+--+:  +--+--+ |
   |  | VM1 |   | VM2 |:  | VM3 | |
   |  +-+   +-+:  +-+ |
   +--+
   Compute Node 1

* R = Router


This picture doesn't look right, there should only be one Router for TenantA 
even with two VMs on a compute node.  You can verify this by looking at how many 
qrouter namespaces are created, but I only see one on my system.



In this idea, SNAT will be created for each tenant.
So, IP consumption of this case is:
  [number of tenant] * [number of compute node]

Therefore, this idea can be reduction in IP consumption than create per router 
per compute node.


This is a huge increase in IP consumption from today though, which is only 
[number of tenants], I'm not sure most deployers have [tenants * compute nodes] 
IPs at their disposal.  And in the worst-case this becomes "Assign a Floating IP 
to all VMs".


-Brian


And, can be avoid security concerns because don't share SNAT between tenant.

I'd like to implement SNAT for Liberty cycle with this idea.

Best regards,
Takanori Miyagishi


-Original Message-
From: Carl Baldwin [mailto:c...@ecbaldwin.net]
Sent: Tuesday, July 07, 2015 2:29 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Neutron] What are problems of Distributed
SNAT?

Hi,

There was some discussion a while back on this subject.  Some alternatives
were captured on etherpad [1] with pros and cons.  Sorry for the delay
in responding.  The initial implementation of DVR went with centralized
SNAT to reduce the scope of the effort and because of a lack consensus
around which alternative to choose.

Carl

[1] https://etherpad.openstack.org/p/decentralized-snat

On Fri, Jun 26, 2015 at 5:02 AM, Miyagishi, Takanori
 wrote:

Hi all,

I'm Takanori Miyagishi.
I and Yushiro Furukawa, my co-worker, are planning to implement

Distributed SNAT in Liberty cycle.

So, I looking for information about Distributed SNAT implementation.
For now, I got some information from openstack-dev ML[1][2][3] and

etherpad[4].

Would you please let me know if you have any other information.

Best regards,
Takanori Miyagishi

[1] Fwd: [Neutron][DVR]Neutron distributed SNAT:



http://lists.openstack.org/pipermail/openstack-dev/2015-February/0564
1

5.html
[2] About DVR limit:



http://lists.openstack.org/pipermail/openstack-dev/2015-January/05440
7

.html
[3] [neutron] dvr l3_snat:



http://lists.openstack.org/pipermail/openstack-dev/2014-November/0498
9

3.html [4] https://etherpad.openstack.org/p/YVR-nova-network



_
_

 OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_
_
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@list

Re: [openstack-dev] [Neutron] Proposing Cedric Brandily to Neutron Core Reviewer Team

[Brian Haley]

+1

On 07/15/2015 02:47 PM, Carl Baldwin wrote:

As the Neutron L3 Lieutenant along with Kevin Benton for control
plane, and Assaf Muller for testing, I would like to propose Cedric
Brandily as a member of the Neutron core reviewer team under these
areas of focus.

Cedric has been a long time contributor to Neutron showing expertise
particularly in these areas.  His knowledge and involvement will be
very important to the project.  He is a trusted member of our
community.  He has been reviewing consistently [1][2] and community
feedback that I've received indicates that he is a solid reviewer.

Existing Neutron core reviewers from these areas of focus, please vote
+1/-1 for the addition of Cedric to the team.

Thanks!
Carl Baldwin

[1] https://review.openstack.org/#/q/reviewer:zzelle%2540gmail.com,n,z
[2] http://stackalytics.com/report/contribution/neutron-group/90

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] - Proposing Miguel Angel Ajo for the Control Plane core team

[Brian Haley]

Big +1

On 07/06/2015 06:02 AM, Kevin Benton wrote:

Hello!

As the Lieutenant of the built-in control plane[1], I am proposing to add Miguel
Angel Ajo to the control plane core reviewer team.

His review stats are inline with the other core reviewers[2], and his work on
improving the stability/performance of the agents over the last year has been
important in making the reference implementation reliable.

Existing cores, please vote +1/-1 for his addition to the team.

Cheers!

1. http://docs.openstack.org/developer/neutron/policies/core-reviewers.html
2. http://stackalytics.com/report/contribution/neutron/30
http://stackalytics.com/report/contribution/neutron/60
http://stackalytics.com/report/contribution/neutron/90

--
Kevin Benton



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposing Brian Haley to Neutron L3 Core Reviewer Team

[Brian Haley]

Thanks for all the support, looking forward to helping out more :)

-Brian

On 06/18/2015 07:34 AM, Paul Michali wrote:

Congratulations Brian! Great addition to the L3 team!

On Wed, Jun 17, 2015 at 11:14 PM Edgar Magana mailto:edgar.mag...@workday.com>> wrote:

Congratulations Brian!  Welcome to the team!

Edgar




On 6/17/15, 3:59 PM, "Carl Baldwin" mailto:c...@ecbaldwin.net>> wrote:

 >It has been a week and feedback has been positive and supportive of
 >Brian's nomination.  Welcome to the L3 core reviewer team, Brian.
 >
 >Carl
 >
 >On Wed, Jun 10, 2015 at 1:11 PM, Carl Baldwin mailto:c...@ecbaldwin.net>> wrote:
 >> Folks,
 >>
 >> As the Neutron L3 Lieutenant [1] under the PTL, Kyle, I'd like to
 >> propose Brian Haley as a member of the Neutron L3 core reviewer team.
 >> Brian has been a long time contributor in Neutron showing expertise
 >> particularly in IPv6, iptables, and Linux kernel matters.  His
 >> knowledge and involvement will be very important especially in this
 >> area.  Brian has become a trusted member of our community.  His review
 >> stats [2][3][4] place him comfortably with other Neutron core
 >> reviewers.  He regularly runs proposed patches himself and gives
 >> insightful feedback.  He has shown a lot of interest in the success of
 >> Neutron.
 >>
 >> Existing Neutron core reviewers from the L3 area of focus, please vote
 >> +1/-1 for the addition of Brian to the core reviewer team.
 >> Specifically, I'm looking for votes from Henry, Assaf, and Mark.
 >>
 >> Thanks!
 >> Carl
 >>
 >> [1]

http://docs.openstack.org/developer/neutron/policies/core-reviewers.html#adding-or-removing-core-reviewers
 >> [2]

https://review.openstack.org/#/q/reviewer:%22Brian+Haley+%253Cbrian.haley%2540hp.com%253E%22,n,z
 >> [3] http://stackalytics.com/report/contribution/neutron-group/90
 >> [4] http://stackalytics.com/?user_id=brian-haley&metric=marks
 >
 >__
 >OpenStack Development Mailing List (not for usage questions)
 >Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
 >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] - dnsmasq 'dhcp-authoritative' option broke multiple DHCP servers

[Brian Haley]

On 05/26/2015 01:12 PM, Salvatore Orlando wrote:

 From the bug Kevin reported it seems multiple dhcp agents per network have been
completely broken by the fix for bug #1345947, so a revert of patch [1] (and
stable backports) should probably be the first thing to do - if nothing else
because the original bug has not nearly the same level of severity of the one it
introduced.
Before doing this however, I am wondering why the various instances of dnsmasq
end up returning NAKs. I expect all instances to have the same hosts file, so
they should be able to respond to DHCPDISCOVER/DHCPREQUEST correctly. Is the
dnsmasq log telling us exactly why the authoritative setting is preventing us
from doing so? (this is more of a curiosity in my side)

[1] https://review.openstack.org/#/c/152080/


In the original case, the DHCPREQUEST is for a renew, which is different than 
for an initial request.  If the server does not have a lease entry (which it 
won't after a restart), then it will NAK, which normally just causes the client 
to retry at INIT state.


I had asked on the dnsmasq list about this [1], and the multiple server question 
was the wildcard, my testing didn't see the error described in the new bug 
though.  I guess the first proposed fix of re-populating the lease information 
doesn't seem like such a bad idea any more, but I will reply to my original 
query with the tcpdump information since I'm confused as to why the second dhcp 
agent stepped-in with a NAK at all after originally offering the same address as 
the first dhcp agent [2].


I would agree the best thing to do is revert the stable backports while we work 
on fixing this in the master branch.


-Brian

[1] http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2015q1/009171.html
[2] https://launchpadlibrarian.net/207180476/dhcp_neutron_bug.html



On 26 May 2015 at 06:57, Ihar Hrachyshka mailto:ihrac...@redhat.com>> wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 05/26/2015 04:35 AM, Kevin Benton wrote:
> Hi,
>
> A recent change[1] to pass '--dhcp-authoritative' to dnsmasq has
> caused DHCPNAK messages when multiple agents are scheduled to a
> network [2].
>
> This was back-ported to Icehouse and Juno so we need a fix that is
> compatible with both of them.
>
> I have two fixes for this so far and a third alternative if we
> don't like those.
>
> The first is hacky, but it's only a few-line change.[3] It adds an
> iptables rule that just stops the DHCPNAKs from making it to the
> client. This is clean to back-port but it doesn't protect clients
> that have filtering disabled (e.g. bare metal).
>
> The second persists the DHCP leases to a database.[4] The downside
> to this was always that being rescheduled to another agent would
> mean no entries in the lease file. This approach adds a work-around
> to generate an initial fake lease file based on all of the ports in
> the network.
>
> A third approach that I don't have a patch pushed for yet is very
> similar to the second. When dnsmasq is in the leasefile-ro mode, it
> will call the script passed to --dhcp-script to get a list of
> leases to start with. This script would be built with the same
> logic as the second one. The only difference between the second
> approach is that dnsmasq wouldn't persist leases to a database.
>

Actually, that approach was initially taken for bug 1345947, but then
the patch was abandoned to be replaced with a simpler
- --dhcp-authoritative approach that ended up with unexpected NAKs for
multi agent setup.

See: https://review.openstack.org/#/c/108272/12

Maybe we actually want to restore the work and merge it after
conflicts are resolved and --dhcp-authoritative option is killed; the
patch was almost merged when --dhcp-authoritative suggestion emerged,
so most of nitpicking work should be complete now (though at the same
time, I totally trust our community to find another pile of nits to
work on for the next few weeks!)


That was my thought as well.
However, we should check whether that patch is ok to backport. For instance I
see what it appears to be adding a script:

[2]
https://review.openstack.org/#/c/108272/12/bin/neutron-dhcp-agent-dnsmasq-lease-init


===

Speaking of regression testing... Are full stack tests already
powerful enough for us to invoke multiple DHCP agents and test the
scenario?

Ihar
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQEcBAEBCAAGBQJVZHvHAAoJEC5aWaUY1u57vukIAJLPpQ9O236NYtOaRTzkL7g8
Io1DmF6jyhJYFqfzoFcrFVbNmM0EsNtvMgZIhI8oYINkkoBYMJPoS2a8FvVUpZHw
u/fmdvdbZgJwy4BCAEF0t+R1t1XLo6eTcPp8f3jABzExWyrLoKEbHJ0aWb5xwJ3u
V74HXxo/PVifrNfxsQPn57ZxqgBvl4GSQAFQKE4FX/H81HWRWRuB5a9aC+hkYC9w
7FqXpf+IFCaS7tYdTSqJUa2/bKs268RQGoVqAYEtmVV5pA3OiMsy459rdLcHqqxS
67lryFh1DTMwI77LjDEanXzWIdMhb3t0YZw7ewpBBLl6P/Lh7xob

Re: [openstack-dev] [nova][neutron]Fail to communicate new host when the first host for a new instance fails

[Brian Haley]

On 05/14/2015 05:29 AM, Neil Jerram wrote:

Hi all, this is about a problem I'm seeing with my Neutron ML2 mechanism driver
[1].  I'm expecting to see an update_port_postcommit call to signal that the
binding:host_id for a port is changing, but I don't see that.

The scenario is launching a new instance in a cluster with two compute hosts,
where we've rigged things so that one of the compute hosts will always be chosen
first, but libvirt isn't correctly configured there and hence the instance
launching attempt will fail.  Then Nova tries to use the other compute host
instead, and that mostly works - except that my mechanism driver still thinks
that the new instance's port is still bound to the first compute host.

Is anyone aware of a known problem in this area (in Juno-level code), or where I
could like to start pinning this down in more detail?


We saw something like this before, perhaps:

https://review.openstack.org/#/c/98340/
https://bugs.launchpad.net/nova/+bug/1327124

It's fixed in Kilo only if that's it.

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Success of the IPv6 Subteam - Proposal to disband

[Brian Haley]

On 05/04/2015 08:37 PM, Sean M. Collins wrote:

It is a bittersweet moment - I am proposing that due to the amazing
success that we have had as a subteam, that because we have
accomplished so much, that it makes sense for our team to
disband and re-integrate with other subteams (the L3 subteam
comes to mind) or have items in the on-demand agenda of the main
meeting.

Unless there is any pressing business, I believe that we will not need a
recurring meeting, and tomorrow's meeting is cancelled.

As always, I am in #openstack-neutron and happy to help.


Sean,

Thanks for leading the team, IPv6 is in a much better place now in Kilo!  I'll 
be the first one to buy you a beer (beers?) in Vancouver.


As long as we adopt the Linux kernel mantra of "You can't do the IPv4 work now 
and punt the IPv6 work for later" I'm fine with pushing future IPv6 work into 
the respective L3/L2/etc sub-teams.


-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Deprecating the use_namespaces option - Now's the time to speak up!

[Brian Haley]

On 03/20/2015 02:57 PM, Assaf Muller wrote:
> Hello everyone,
> 
> The use_namespaces option in the L3 and DHCP Neutron agents controls if you
> can create multiple routers and DHCP networks managed by a single L3/DHCP 
> agent,
> or if the agent manages only a single resource.
> 
> Are the setups out there *not* using the use_namespaces option? I'm curious as
> to why, and if it would be difficult to migrate such a setup to use 
> namespaces.

This is a recent Neutron bug where someone is not using namespaces, so they 
exist:

https://bugs.launchpad.net/neutron/+bug/1428007

> I'm asking because use_namespaces complicates Neutron code for what I gather
> is an option that has not been relevant for years. I'd like to deprecate the 
> option
> for Kilo and remove it in Liberty.

+1 from me for deprecation.

-Brian


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] OpenFlow security groups (pre-benchmarking plan)

[Brian Haley]

On 02/25/2015 08:52 AM, Miguel Ángel Ajo wrote:
> I’m writing a plan/script to benchmark OVS+OF(CT) vs OVS+LB+iptables+ipsets,
> so we can make sure there’s a real difference before jumping into any
> OpenFlow security group filters when we have connection tracking in OVS.
> 
> The plan is to keep all of it in a single multicore host, and make all the 
> measures
> within it, to make sure we just measure the difference due to the software 
> layers.
> 
> Suggestions or ideas on what to measure are welcome, there’s an initial draft 
> here:
> 
> https://github.com/mangelajo/ovs-experiments/tree/master/ovs-ct

Thanks for writing this up Miguel.

I realize this is more focusing on performance (how fast the packets flow), but
one of the orthogonal issues to Security Groups in general is the time it takes
for Neutron to apply or update them, for example, iptables_manager.apply().  I
would like to make sure that time doesn't grow any larger than it is today.
This can probably all be scraped from log files, so wouldn't require any special
work, except for testing with a large SG set.

Thanks,

-Brian


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] IPv6 Status

[Brian Haley]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/05/2015 05:38 AM, Ihar Hrachyshka wrote:
> On 02/05/2015 09:14 AM, Andreas Scheuring wrote:
>> Hi,
> 
>> is there a central place where I can find a matrix (or something similar)
>> that shows what is currently supposed to work in the sense of IPv6
>> Networking?

I'm not sure there's a matrix, but there are a number of updates coming in
Kilo, check https://blueprints.launchpad.net/neutron/kilo for the IPv6 ones.

Most of the bug fixes are tagged and can be found here:

https://bugs.launchpad.net/neutron/+bugs?field.tag=ipv6

>> I also had a look at a couple of blueprints out there, but I'm looking
>> for a simple overview containing what's supported, on which features are
>> people working on and what's future. I mean all the good stuff for Tenant
>> Networks like
> 
>> - SNAT - FloatingIP - External Provider Networks - DVR - fwaas, 
>> vpnaas,...

There will be no default SNATv6 or Floating IPv6 as decided at the Kilo Summit
- - they're really not necessary if the VM has a global address.

Probably the best thing for you to do is spin-up a devstack with IPv6 enabled
and see how things work, putting these in local.conf will enable both DVR and
IPv6 (assuming you have RECLONE=yes):

Q_DVR_MODE=dvr_snat

# to enable IPv4 and IPv6
IP_VERSION=4+6

>> and also about the Host Network - e.g. vxlan/gre tunneling via ipv6 host
>> network...
> 
> AFAIK it's not supported by OVS yet. The last time I talked to a guy who
> worked on the feature, he told me that it will take some time, and it will
> probably be VXLAN only. (Unless someone steps in.)

OVS/VXLAN w/IPv6 is not supported, pointer to thread:

http://openvswitch.org/pipermail/discuss/2015-January/016344.html

- -Brian
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iQEcBAEBAgAGBQJU05XhAAoJEIYQqpVulyUozYgH/iple8dvRoyE0Z+9aUObcYxi
uh5jkly4zeCs7OUEOlMDeAsboKBdgEc+kSLzu6ianqd1f8CtHXG1iROn2YTb7z05
HN5bPByT9pZW5eu5lSN0KsOvlnpJCvK/Kz6xbW+cJJ03YCmHZkto64SEOcnJJ1iq
mFFnKOjxDlxHUGJyt0GNto7hQNV9RUSVA6fk7omsn5UnSP4RcZJjqbXCFW4mmU9j
ppUGB+dnGeQyKq0egPN9bzNRudSfVKA6mne5ipxOhYNzju5LBp84xqIAlBq0w7k7
8SIYsdDIrmPjWwckr6+39QHFSCUPqbNmRyH2H28CoYpd7ZQuvPZ2osseNX0AahA=
=ahi+
-END PGP SIGNATURE-

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] high dhcp lease times in neutron deployments considered harmful (or not???)

[Brian Haley]

On 02/03/2015 05:10 AM, Kevin Benton wrote:
>>The unicast DHCP will make it to the "wire", but if you've renumbered the
> subnet either a) the DHCP server won't respond because it's IP has changed as
> well; or b) the DHCP server won't respond because there is no mapping for the 
> VM
> on it's old subnet.
> 
> We aren't changing the DHCP server's IP here. The process that I saw was to 
> add
> a subnet and start moving VMs over. It's not 'b' either, because the server
> generates a DHCPNAK in response and which will immediately cause the client to
> release/renew. I have verified this behavior already and recorded a packet
> capture for you.[1] 
> 
> In the capture, the renewal value is 4 seconds. I captured one renewal before
> the IP address change from 99.99.99.5 to 10.0.0.25 took place. You can see on
> the next renewal, the DHCP server immediately generates a NACK. The client 
> then
> releases its address, requests a new one, assigns it and ACKs within a couple 
> of
> seconds. 

Thanks for the trace.  So one thing I noticed is that this unicast DHCP only got
to the server since you created a second subnet on this network (dest MAC of
packet was that of same router interface).  If you had created a second network
and subnet this would have been dropped (different broadcast domain).  These
little differences are things users need to know because they lead to heads
banging on desks :(

>>This would happen if the AZ their VM was in went offline as well, at which
> point they would change their design to be more cloud-aware than it was.  
> Let's
> not heap all the blame on neutron - the user is tasked with vetting that
> their decisions meet the requirements they desire by thoroughly testing it.
> 
> An availability zone going offline is not the same as an API operation that
> takes a day to apply. In an internal cloud, maintenance for AZs can be
> advertised and planned around by tenants running single-AZ services. Even if 
> you
> want to reference a public cloud, look how much of the Internet breaks when
> Amazon's us-east-1a or us-east-1d AZs have issues. Even though people are
> supposed to be bringing cattle to the cloud, a huge portion already have pets
> that they are attached to or that they can't convert into cattle. 

You completely missed the context of my reply Kevin - an AZ failure is not a
planned event.  You said people bring pets along, and rebooting them is painful.
 I said that's a bad design because other things can cause it to go offline, for
example:

1. Compute node failure
2. Network node failure
3. Router/switch failure
4. Internet failure
...
99. API call

All the user knows is they can't reach their VM - the cause doesn't matter when
they can't sell their widgets to customers because their site is down.  If it
takes 10 minutes for them to re-create their instance elsewhere that cannot be
blamed on neutron, even if it was our API call that caused it to go offline.

> If our floating IP 'associate' action took 12 hours to take effect on a 
> running
> instance, would telling users to reboot their instances to apply floating IPs
> faster be okay? I would certainly heap the blame on Neutron there.

The difference in a port IP change API call is that it requires action on the
VMs part that neutron can't trigger immediately.  It's still asynchronous like a
floating IP call, but the delay is typically going to be longer.  All we can say
is it will take from (0 -> interval) seconds.  How is warning the user about
this a bad thing?

>>How about a big (*) next to all the things that could cause issues?  :)
> 
> You want to put it next to all of the API calls to put the burden on the 
> users.
> I want to put it next to the DHCP renewal interval in the config files to put
> the burden on the operators. :)
> 
> (*) Increasing this value will increase the delay between API calls and when
> they take effect on the data plane for any that depend on DHCP to relay the
> information. (e.g. port IP/subnet changes, port dhcp option changes, subnet
> gateways, subnet routes, subnet DNS servers, etc)

There is no delay in the API call here, the port was updated just as the user
requested.  Since they can't see into my config file (unless they look at their
lease info or run a tcpdump trace) they are essentially making a blind change
that immediately affects their instance.

And adding a DHCP option to tell them to renew more frequently doesn't fix the
problem, it only lessens it to ~(interval/2) - that might not be acceptable to
users and they need to know the danger.  This is the one point I've been trying
to get across in this whole discussion - these are advanc

Re: [openstack-dev] [neutron] high dhcp lease times in neutron deployments considered harmful (or not???)

[Brian Haley]

quired to re-gain connectivity.

> In summary, the information the DHCP server gives to clients is not static. 
> Unless we eliminate updates to everything in the Neutron API that results in 
> different DHCP lease information, my suggestion is that we include a new
> option for the renewal interval and have the default set <5 minutes. We can
> leave the lease default to 1 day so the amount of time a DHCP server can be
> offline without impacting running clients can stay the same.

I'm fine with adding Option 58, even though it only lessens the effect of this
problem, doesn't truly fix it, and might not work with all clients (like in 
Cirros).

-Brian

> On Fri, Jan 30, 2015 at 8:00 AM, Brian Haley  <mailto:brian.ha...@hp.com>> wrote:
> 
> Kevin,
> 
> The only thing this discussion has convinced me of is that allowing users to 
> change the fixed IP address on a neutron port leads to a bad
> user-experience. Even with an 8-minute renew time you're talking up to a
> 7-minute blackout (87.5% of lease time before using broadcast).  This is time
> that customers are paying for.  Most would have rebooted long before then,
> true?  Cattle not pets, right?
> 
> Changing the lease time is just papering-over the real bug - neutron doesn't 
> support seamless changes in IP addresses on ports, since it totally relies
> on the dhcp configuration settings a deployer has chosen.  Bickering over the
> lease time doesn't fix that non-deterministic recovery for the VM.
> Documenting a VM reboot is necessary, or even deprecating this (you won't
> like that) are sounding better to me by the minute.
> 
> Is there anyone else that has used, or has customers using, this part of the 
> neutron API?  Can they share their experiences?
> 
> -Brian
> 
> 
> On 01/30/2015 07:26 AM, Kevin Benton wrote:
>>> But they will if we document it well, which is what Salvatore suggested.
>> 
>> I don't think this is a good approach, and it's a big part of why I
> started this
>> thread. Most of the deployers/operators I have worked with only read the
>> bare minimum documentation to get a Neutron deployment working and they
>> only adjust the settings necessary for basic functionality.
>> 
>> We have an overwhelming amount of configuration options and adding a note 
>> specifying that a particular setting for DHCP leases has been optimized to 
>> reduce logging at the cost of long downtimes during port IP address
> updates is a
>> waste of time and effort on our part.
>> 
>>> I think the current default value is also more indicative of something
>> you'd find in your house, or at work - i.e. stable networks.
>> 
>> Tenants don't care what the DHCP lease time is or that it matches what
>> they would see from a home router. They only care about connectivity.
>> 
>>> One solution is to disallow this operation.
>> 
>> I want this feature to be useful in deployments by default, not strip it 
>> away. You can probably do this with /etc/neutron/policy.json without a
>> code change if you wanted to block it in a deployment like yours where you
>> have
> such
>> a high lease time.
>> 
>>> Perhaps letting the user set it, but allow the admin to set the valid
>>> range
>> for min/max?  And if they don't specify they get the default?
>> 
>> Tenants wouldn't have any reason to adjust this default. They would be
> even less
>> likely than the operator to know about this weird relationship between a
>> DHCP setting and the amount of time they lose connectivity after updating
>> their ports' IPs.
>> 
>>> It impacts anyone that hasn't changed from the default since July 2013
>>> and
> later
>> (Havana), since if they don't notice, they might get bitten by it.
>> 
>> Keep in mind that what I am suggesting with the lease-renewal-time would
>> be separate from the lease expiration time. The only difference that an
>> operator would see on upgrade (if using the defaults) is increased DHCP
>> traffic and
> more
>> logs to syslog from dnsmasq. The lease time would still be the same so the 
>> downtime windows for DHCP agents would be maintained. That is much less of
>> an impact than many of the non-config changes we make between cycles.
>> 
>> To clarify, even with an option for dhcp-renewal-time I am proposing, you
>> are still opposed to setting it to anything low because of logging and the
>> ~24 bps background DHCP traffic per VM?
>> 
>> On Thu, Jan 29, 2015 at 7:11 PM, Brian Haley  <mailto:brian.ha...@hp.com>
>> <mailto:bri

Re: [openstack-dev] [neutron] high dhcp lease times in neutron deployments considered harmful (or not???)

[Brian Haley]

Kevin,

The only thing this discussion has convinced me of is that allowing users to
change the fixed IP address on a neutron port leads to a bad user-experience.
Even with an 8-minute renew time you're talking up to a 7-minute blackout (87.5%
of lease time before using broadcast).  This is time that customers are paying
for.  Most would have rebooted long before then, true?  Cattle not pets, right?

Changing the lease time is just papering-over the real bug - neutron doesn't
support seamless changes in IP addresses on ports, since it totally relies on
the dhcp configuration settings a deployer has chosen.  Bickering over the lease
time doesn't fix that non-deterministic recovery for the VM.  Documenting a VM
reboot is necessary, or even deprecating this (you won't like that) are sounding
better to me by the minute.

Is there anyone else that has used, or has customers using, this part of the
neutron API?  Can they share their experiences?

-Brian


On 01/30/2015 07:26 AM, Kevin Benton wrote:
>>But they will if we document it well, which is what Salvatore suggested.
> 
> I don't think this is a good approach, and it's a big part of why I started 
> this
> thread. Most of the deployers/operators I have worked with only read the bare
> minimum documentation to get a Neutron deployment working and they only adjust
> the settings necessary for basic functionality.
> 
> We have an overwhelming amount of configuration options and adding a note
> specifying that a particular setting for DHCP leases has been optimized to
> reduce logging at the cost of long downtimes during port IP address updates 
> is a
> waste of time and effort on our part. 
> 
>>I think the current default value is also more indicative of something
> you'd find in your house, or at work - i.e. stable networks.
> 
> Tenants don't care what the DHCP lease time is or that it matches what they
> would see from a home router. They only care about connectivity. 
> 
>>One solution is to disallow this operation.
> 
> I want this feature to be useful in deployments by default, not strip it
> away. You can probably do this with /etc/neutron/policy.json without a code
> change if you wanted to block it in a deployment like yours where you have 
> such
> a high lease time.
> 
>>Perhaps letting the user set it, but allow the admin to set the valid range
> for min/max?  And if they don't specify they get the default?
> 
> Tenants wouldn't have any reason to adjust this default. They would be even 
> less
> likely than the operator to know about this weird relationship between a DHCP
> setting and the amount of time they lose connectivity after updating their
> ports' IPs.
> 
>>It impacts anyone that hasn't changed from the default since July 2013 and 
>>later
> (Havana), since if they don't notice, they might get bitten by it.
> 
> Keep in mind that what I am suggesting with the lease-renewal-time would be
> separate from the lease expiration time. The only difference that an operator
> would see on upgrade (if using the defaults) is increased DHCP traffic and 
> more
> logs to syslog from dnsmasq. The lease time would still be the same so the
> downtime windows for DHCP agents would be maintained. That is much less of an
> impact than many of the non-config changes we make between cycles.
> 
> To clarify, even with an option for dhcp-renewal-time I am proposing, you are
> still opposed to setting it to anything low because of logging and the ~24 bps
> background DHCP traffic per VM?
> 
> On Thu, Jan 29, 2015 at 7:11 PM, Brian Haley  <mailto:brian.ha...@hp.com>> wrote:
> 
> On 01/29/2015 05:28 PM, Kevin Benton wrote:
> >>How is Neutron breaking this?  If I move a port on my physical switch 
> to a
> > different subnet, can you still communicate with the host sitting on it?
> > Probably not since it has a view of the world (next-hop router) that no 
> longer
> > exists, and the network won't route packets for it's old IP address to 
> the new
> > location.  It has to wait for it's current DHCP lease to tick down to 
> the point
> > where it will use broadcast to get a new one, after which point it will 
> work.
> >
> > That's not just moving to a different subnet. That's moving to a 
> different
> > broadcast domain. Neutron supports multiple subnets per network 
> (broadcast
> > domain). An address on either subnet will work. The router has two 
> interfaces
> > into the network, one on each subnet.[2]
> >
> >
> >>Does it work on Windows VMs too?  People run those in clouds too.  The 
> point is
&g

Re: [openstack-dev] [neutron] high dhcp lease times in neutron deployments considered harmful (or not???)

[Brian Haley]

On 01/29/2015 05:28 PM, Kevin Benton wrote:
>>How is Neutron breaking this?  If I move a port on my physical switch to a
> different subnet, can you still communicate with the host sitting on it?
> Probably not since it has a view of the world (next-hop router) that no longer
> exists, and the network won't route packets for it's old IP address to the new
> location.  It has to wait for it's current DHCP lease to tick down to the 
> point
> where it will use broadcast to get a new one, after which point it will work.
> 
> That's not just moving to a different subnet. That's moving to a different
> broadcast domain. Neutron supports multiple subnets per network (broadcast
> domain). An address on either subnet will work. The router has two interfaces
> into the network, one on each subnet.[2]
> 
> 
>>Does it work on Windows VMs too?  People run those in clouds too.  The point 
>>is
> that if we don't know if all the DHCP clients will support it then it's a
> non-starter since there's no way to tell from the server side.
> 
> It appears they do.[1] Even for clients that don't, the worst case scenario is
> just that they are stuck where we are now.
> 
>>"... then the deployer can adjust the value upwards...", hmm, can they adjust 
>>it
> downwards as well?  :)
> 
> Yes, but most people doing initial openstack deployments don't and wouldn't
> think to without understanding the intricacies of the security groups 
> filtering
> in Neutron.

But they will if we document it well, which is what Salvatore suggested.

>>I'm glad you're willing to "boil the ocean" to try and get the default 
>>changed,
> but is all this really worth it when all you have to do is edit the config 
> file
> in your deployment?  That's why the value is there in the first place.
> 
> The default value is basically incompatible with port IP changes. We shouldn't
> be shipping defaults that lead to half-broken functionality. What I'm
> understanding is that the current default value is to workaround shortcomings 
> in
> dnsmasq. This is an example of implementation details leaking out and leading 
> to
> bad UX. 

I think the current default value is also more indicative of something you'd
find in your house, or at work - i.e. stable networks.

I had another thought on this Kevin, hoping that we could come to some
resolution, because sure, shipping broken functionality isn't great.  But here's
the rub - how do we make a change in a fixed IP work in *all* deployments?
Since the end-user can't set this value, they'll run into this problem in my
deployment, or any other that has some not-very-short lease time.  One solution
is to disallow this operation.  The other is to fix neutron to make this work
better (I don't know what that involves, but there's bound to be a way).
Perhaps letting the user set it, but allow the admin to set the valid range for
min/max?  And if they don't specify they get the default?

> If we had an option to configure how often iptables rules were refreshed to
> match their security group, there is no way we would have a default of 12 
> hours.
> This is essentially the same level of connectivity interruption, it just 
> happens
> to be a narrow use case so it hasn't been getting any attention.
> 
> To flip your question around, why do you care if the default is lower? You
> already adjust it beyond the 1 day default in your deployment, so how would a
> different default impact you?

It impacts anyone that hasn't changed from the default since July 2013 and later
(Havana), since if they don't notice, they might get bitten by it.

-Brian


> 
> 1. http://support.microsoft.com/kb/121005
> 2. Similar to using the "secondary" keyword on Cisco devices. Or just the "ip
> addr add" command on linux.
> 
> On Thu, Jan 29, 2015 at 1:34 PM, Brian Haley  <mailto:brian.ha...@hp.com>> wrote:
> 
> On 01/29/2015 03:55 AM, Kevin Benton wrote:
> >>Why would users want to change an active port's IP address anyway?
> >
> > Re-addressing. It's not common, but the entire reason I brought this up 
> is
> > because a user was moving an instance to another subnet on the same 
> network and
> > stranded one of their VMs.
> >
> >> I worry about setting a default config value to handle a very unusual 
> use case.
> >
> > Changing a static lease is something that works on normal networks so I 
> don't
> > think we should break it in Neutron without a really good reason.
> 
> How is Neutron breaking this?  If I move a port 

Re: [openstack-dev] [neutron] high dhcp lease times in neutron deployments considered harmful (or not???)

[Brian Haley]

On 01/29/2015 03:55 AM, Kevin Benton wrote:
>>Why would users want to change an active port's IP address anyway?
> 
> Re-addressing. It's not common, but the entire reason I brought this up is
> because a user was moving an instance to another subnet on the same network 
> and
> stranded one of their VMs.
> 
>> I worry about setting a default config value to handle a very unusual use 
>> case.
> 
> Changing a static lease is something that works on normal networks so I don't
> think we should break it in Neutron without a really good reason.

How is Neutron breaking this?  If I move a port on my physical switch to a
different subnet, can you still communicate with the host sitting on it?
Probably not since it has a view of the world (next-hop router) that no longer
exists, and the network won't route packets for it's old IP address to the new
location.  It has to wait for it's current DHCP lease to tick down to the point
where it will use broadcast to get a new one, after which point it will work.

> Right now, the big reason to keep a high lease time that I agree with is that 
> it
> buys operators lots of dnsmasq downtime without affecting running clients. To
> get the best of both worlds we can set DHCP option 58 (a.k.a dhcp-renewal-time
> or T1) to 240 seconds. Then the lease time can be left to be something large
> like 10 days to allow for tons of DHCP server downtime without affecting 
> running
> clients.
> 
> There are two issues with this approach. First, some simple dhcp clients don't
> honor that dhcp option (e.g. the one with Cirros), but it works with dhclient 
> so
> it should work on CentOS, Fedora, etc (I verified it works on Ubuntu). This
> isn't a big deal because the worst case is what we have already (half of the
> lease time). The second issue is that dnsmasq hardcodes that option, so a 
> patch
> would be required to allow it to be specified in the options file. I am happy 
> to
> submit the patch required there so that isn't a big deal either.

Does it work on Windows VMs too?  People run those in clouds too.  The point is
that if we don't know if all the DHCP clients will support it then it's a
non-starter since there's no way to tell from the server side.

> If we implement that fix, the remaining issue is Brian's other comment about 
> too
> much DHCP traffic. I've been doing some packet captures and the standard
> request/reply for a renewal is 2 unicast packets totaling about 725 bytes.
> Assuming 10,000 VMs renewing every 240 seconds, there will be an average of 
> 242
> kbps background traffic across the entire network. Even at a density of 50 
> VMs,
> that's only 1.2 kbps per compute node. If that's still too much, then the
> deployer can adjust the value upwards, but that's hardly a reason to have a 
> high
> default.

"... then the deployer can adjust the value upwards...", hmm, can they adjust it
downwards as well?  :)

> That just leaves the logging problem. Since we require a change to dnsmasq
> anyway, perhaps we could also request an option to suppress logs from 
> renewals?
> If that's not adequate, I think 2 log entries per vm every 240 seconds is 
> really
> only a concern for operators with large clouds and they should have the
> knowledge required to change a config file anyway. ;-)

I'm glad you're willing to "boil the ocean" to try and get the default changed,
but is all this really worth it when all you have to do is edit the config file
in your deployment?  That's why the value is there in the first place.

Sorry, I'm still unconvinced we need to do anything more than document this.

-Brian



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] high dhcp lease times in neutron deployments considered harmful (or not???)

[Brian Haley]

Hi Kevin,

On 01/28/2015 03:50 AM, Kevin Benton wrote:
> Hi,
> 
> Approximately a year and a half ago, the default DHCP lease time in Neutron 
> was
> increased from 120 seconds to 86400 seconds.[1] This was done with the goal of
> reducing DHCP traffic with very little discussion (based on what I can see in
> the review and bug report). While it it does indeed reduce DHCP traffic, I 
> don't
> think any bug reports were filed showing that a 120 second lease time resulted
> in too much traffic or that a jump all of the way to 86400 seconds was 
> required
> instead of a value in the same order of magnitude.
> 
> Why does this matter? 
> 
> Neutron ports can be updated with a new IP address from the same subnet or
> another subnet on the same network. The port update will result in 
> anti-spoofing
> iptables rule changes that immediately stop the old IP address from working on
> the host. This means the host is unreachable for 0-12 hours based on the 
> current
> default lease time without manual intervention[2] (assuming half-lease length
> DHCP renewal attempts).

So I'll first comment on the problem.  You're essentially "pulling the rug" out
from under these VMs by changing their IP (and that of their router and DHCP/DNS
server), but you expect they should fail quickly and come right back online.  In
a non-Neutron environment wouldn't the IT person that did this need some pretty
good heat-resistant pants for all the flames from pissed-off users?  Sure, the
guy on his laptop will just bounce the connection, but servers (aka VMs) should
stay pretty static.  VMs are servers (and cows according to some).

The correct solution is to be able to renumber the network so there is no issue
with the anti-spoofing rules dropping packets, or the VMs having an unreachable
IP address, but that's a much bigger nut to crack.

> Why is this on the mailing list?
> 
> In an attempt to make the VMs usable in a much shorter timeframe following a
> Neutron port address change, I submitted a patch to reduce the default DHCP
> lease time to 8 minutes.[3] However, this was upsetting to several people,[4] 
> so
> it was suggested I bring this discussion to the mailing list. The following 
> are
> the high-level concerns followed by my responses:
> 
>   * 8 minutes is arbitrary
>   o Yes, but it's no more arbitrary than 1440 minutes. I picked it as an
> interval because it is still 4 times larger than the last short value,
> but it still allows VMs to regain connectivity in <5 minutes in the
> event their IP is changed. If someone has a good suggestion for 
> another
> interval based on known dnsmasq QPS limits or some other quantitative
> reason, please chime in here.

We run 48 hours as the default in our public cloud, and I did some digging to
remind myself of the multiple reasons:

1. Too much DHCP traffic.  Sure, only that initial request is broadcast, but
dnsmasq is very verbose and loves writing to syslog for everything it does -
less is more.  Do a scale test with 10K VMs and you'll quickly find out a large
portion of traffic is DHCP RENEWs, and syslog is huge.

2. During a control-plane upgrade or outage, having a short DHCP lease time will
take all your VMs offline.  The old value of 2 minutes is not a realistic value
for an upgrade, and I don't think 8 minutes is much better.  Yes, when DHCP is
down you can't boot a new VM, but as long as customers can get to their existing
VMs they're pretty happy and won't scream bloody murder.

There's probably more, but those were the top two, with #2 being most important.

>   * other datacenters use long lease times
>   o This is true, but it's not really a valid comparison. In most regular
> datacenters, updating a static DHCP lease has no effect on the data
> plane so it doesn't matter that the client doesn't react for 
> hours/days
> (even with DHCP snooping enabled). However, in Neutron's case, the
> security groups are immediately updated so all traffic using the old
> address is blocked.

Yes, and choosing the lease time is a deployment decision that needs to take a
lot of things into account.  Like I said, we don't even use the default.  The
default should just be a good guess for a standard deployment, not a value that
caters towards the edge cases, especially when the value is tunable in 
neutron.conf.

>   * dhcp traffic is scary because it's broadcast
>   o ARP traffic is also broadcast and many clients will expire entries 
> every
> 5-10 minutes and re-ARP. L2population may be used to prevent ARP
> propagation, so the comparison between DHCP and ARP isn't always
> relevant here.

I don't recall anyone being scared of broadcast, and can't find any comments
regarding it in https://review.openstack.org/#/c/150595/

> Please reply back with your opinions/anecdotes/data related to short DHCP 
> lease
> times.

I can only speculate on why 24 hours was chosen as

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Brian Haley]

On 01/22/2015 02:35 PM, Kevin Benton wrote:
> Right, there are two bugs here. One is in whatever went wrong with defer_apply
> and one is with this exception handling code. I would allow the fix to go in 
> for
> the exception handling and then file another bug for the actual underlying
> defer_apply bug.

What went wrong with defer_apply() was caused by oslo.concurrency - version
1.4.1 seems to fix the problem, see https://review.openstack.org/#/c/149400/
(thanks Ihar!)

Xavier - can you update your oslo.concurrency to that version and verify it
helps?  It seems to work in my config.

Then the change in the other patchset could be applied, along with a test that
triggers exceptions so this gets caught.

Thanks,

-Brian

> On Thu, Jan 22, 2015 at 10:32 AM, Brian Haley  <mailto:brian.ha...@hp.com>> wrote:
> 
> On 01/22/2015 01:06 PM, Kevin Benton wrote:
> > There was a bug for this already.
> > https://bugs.launchpad.net/bugs/1413111
> 
> Thanks Kevin.  I added more info to it, but don't think the patch 
> proposed there
> is correct.  Something in the iptables manager defer_apply() code isn't
> quite right.
> 
> -Brian
> 
> 
> > On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley  <mailto:brian.ha...@hp.com>
> > <mailto:brian.ha...@hp.com <mailto:brian.ha...@hp.com>>> wrote:
> >
> > On 01/22/2015 10:17 AM, Carl Baldwin wrote:
> > > I think this warrants a bug report.  Could you file one with what 
> you
> > > know so far?
> >
> > Carl,
> >
> > Seems as though a recent change introduced a bug.  This is on a 
> devstack
> > I just created today, at l3/vpn-agent startup:
> >
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback 
> (most
> > recent call last):
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> > "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return
> > func(*args, **kwargs)
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> > "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
> process_router
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> >  self._process_external(ri)
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> > "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
> _process_external
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> >  self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> UnboundLocalError:
> > local variable 'existing_floating_ips' referenced before assignment
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> > Traceback (most recent call last):
> >   File 
> "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py",
> line
> > 82, in _spawn_n_impl
> > func(*args, **kwargs)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1093, in
> > _process_router_update
> > self._process_router_if_compatible(router)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1047, in
> > _process_router_if_compatible
> > self._process_added_router(router)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1056, in
> > _process_added_router
> > self.process_router(ri)
> >   File "/opt/stack/neutron/neutron/common/utils.py", line 345, in 
> call
> > self.logger(e)
> >   File
> "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line
> > 82, in __exit__
> > six.reraise(self.type_, self.value, self.tb)
> >   File "/opt/stack/neutron/neutron/common/utils.py", line 342, in 
> call
> > return func(*args, **kwargs)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
> > process_router
> > self._process_external(ri)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
> > _process_external
> > self._update_fip_statuses(ri,

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Brian Haley]

On 01/22/2015 01:06 PM, Kevin Benton wrote:
> There was a bug for this already.
> https://bugs.launchpad.net/bugs/1413111

Thanks Kevin.  I added more info to it, but don't think the patch proposed there
is correct.  Something in the iptables manager defer_apply() code isn't quite 
right.

-Brian


> On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley  <mailto:brian.ha...@hp.com>> wrote:
> 
> On 01/22/2015 10:17 AM, Carl Baldwin wrote:
> > I think this warrants a bug report.  Could you file one with what you
> > know so far?
> 
> Carl,
> 
> Seems as though a recent change introduced a bug.  This is on a devstack
> I just created today, at l3/vpn-agent startup:
> 
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most
> recent call last):
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return
> func(*args, **kwargs)
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in 
> process_router
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   
>  self._process_external(ri)
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in 
> _process_external
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   
>  self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent 
> UnboundLocalError:
> local variable 'existing_floating_ips' referenced before assignment
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", 
> line
> 82, in _spawn_n_impl
> func(*args, **kwargs)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1093, in
> _process_router_update
> self._process_router_if_compatible(router)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1047, in
> _process_router_if_compatible
> self._process_added_router(router)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1056, in
> _process_added_router
> self.process_router(ri)
>   File "/opt/stack/neutron/neutron/common/utils.py", line 345, in call
> self.logger(e)
>   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", 
> line
> 82, in __exit__
> six.reraise(self.type_, self.value, self.tb)
>   File "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
> return func(*args, **kwargs)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
> process_router
> self._process_external(ri)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
> _process_external
> self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
> UnboundLocalError: local variable 'existing_floating_ips' referenced 
> before
> assignment
> 
> Since that's happening while we're holding the iptables lock I'm assuming
> no rules are being applied.
> 
> I'm looking into it now, will file a bug if there isn't already one.
> 
> -Brian


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Brian Haley]

On 01/22/2015 10:17 AM, Carl Baldwin wrote:
> I think this warrants a bug report.  Could you file one with what you
> know so far?

Carl,

Seems as though a recent change introduced a bug.  This is on a devstack
I just created today, at l3/vpn-agent startup:

2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most 
recent call last):
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File 
"/opt/stack/neutron/neutron/common/utils.py", line 342, in call
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return 
func(*args, **kwargs)
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File 
"/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in process_router
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent 
self._process_external(ri)
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File 
"/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in _process_external
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent 
self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent UnboundLocalError: 
local variable 'existing_floating_ips' referenced before assignment
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 82, 
in _spawn_n_impl
func(*args, **kwargs)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1093, in 
_process_router_update
self._process_router_if_compatible(router)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1047, in 
_process_router_if_compatible
self._process_added_router(router)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1056, in 
_process_added_router
self.process_router(ri)
  File "/opt/stack/neutron/neutron/common/utils.py", line 345, in call
self.logger(e)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 
82, in __exit__
six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
return func(*args, **kwargs)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in 
process_router
self._process_external(ri)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in 
_process_external
self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
UnboundLocalError: local variable 'existing_floating_ips' referenced before 
assignment

Since that's happening while we're holding the iptables lock I'm assuming
no rules are being applied.

I'm looking into it now, will file a bug if there isn't already one.

-Brian


> On Wed, Jan 21, 2015 at 2:24 PM, Brian Haley  wrote:
>> On 01/21/2015 02:29 PM, Xavier León wrote:
>>> On Tue, Jan 20, 2015 at 10:32 PM, Brian Haley  wrote:
>>>> On 01/20/2015 09:20 AM, Xavier León wrote:
>>>>> Hi all,
>>>>>
>>>>> we've been doing some tests with openstack kilo and found
>>>>> out a problem: iptables routes are not being injected to the
>>>>> router namespace.
>>>>>
>>>>> Scenario:
>>>>> - a private network NOT connected to the outside world.
>>>>> - a router with only one interface connected to the private network.
>>>>> - a vm instance connected to the private network as well.
>> 
>>>> Are you sure the l3-agent is running?  You should have seen wrapped rules 
>>>> from
>>>> it in most of these tables, for example:
>>>>
>>>> # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015
>>>> *filter
>>>> :INPUT ACCEPT [34:10882]
>>>> :FORWARD ACCEPT [0:0]
>>>> :OUTPUT ACCEPT [1:84]
>>>> :neutron-filter-top - [0:0]
>>>> :neutron-l3-agent-FORWARD - [0:0]
>>>> :neutron-l3-agent-INPUT - [0:0]
>>>> :neutron-l3-agent-OUTPUT - [0:0]
>>>> :neutron-l3-agent-local - [0:0]
>>>> [...]
>>>
>>> Yes, the l3-agent is up and running. I see these rules when executing
>>> the same test in juno but not in kilo. FYI, it's a all-in-one devstack
>>> deployment.
>>>
>>>>
>>>> I would check the log files for any errors.
>>>
>>> There are no errors in the logs.
>>>
>>> After digging a bit more, we have seen that setting the config value
>>> of enable_isolated_metadata to True (default: False) in dhcp_agent.ini
>>> solves the problem in our scenario.
>>> However, this c

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Brian Haley]

On 01/21/2015 02:29 PM, Xavier León wrote:
> On Tue, Jan 20, 2015 at 10:32 PM, Brian Haley  wrote:
>> On 01/20/2015 09:20 AM, Xavier León wrote:
>>> Hi all,
>>>
>>> we've been doing some tests with openstack kilo and found
>>> out a problem: iptables routes are not being injected to the
>>> router namespace.
>>>
>>> Scenario:
>>> - a private network NOT connected to the outside world.
>>> - a router with only one interface connected to the private network.
>>> - a vm instance connected to the private network as well.

>> Are you sure the l3-agent is running?  You should have seen wrapped rules 
>> from
>> it in most of these tables, for example:
>>
>> # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015
>> *filter
>> :INPUT ACCEPT [34:10882]
>> :FORWARD ACCEPT [0:0]
>> :OUTPUT ACCEPT [1:84]
>> :neutron-filter-top - [0:0]
>> :neutron-l3-agent-FORWARD - [0:0]
>> :neutron-l3-agent-INPUT - [0:0]
>> :neutron-l3-agent-OUTPUT - [0:0]
>> :neutron-l3-agent-local - [0:0]
>> [...]
> 
> Yes, the l3-agent is up and running. I see these rules when executing
> the same test in juno but not in kilo. FYI, it's a all-in-one devstack
> deployment.
> 
>>
>> I would check the log files for any errors.
> 
> There are no errors in the logs.
> 
> After digging a bit more, we have seen that setting the config value
> of enable_isolated_metadata to True (default: False) in dhcp_agent.ini
> solves the problem in our scenario.
> However, this change in configuration was not necessary before (our
> tests passed in juno for that matter with that setting to False). So
> we were wondering if there has been a change in how the metadata
> service is accessed in such scenarios, a new issue because of the l3
> agent refactoring or any other problem in our setup we haven't
> narrowed yet.

There have been some changes recently in the code, perhaps:

https://review.openstack.org/#/c/135467/

Or just look at some of the other recent changes in the repository?

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Brian Haley]

On 01/20/2015 09:20 AM, Xavier León wrote:
> Hi all,
> 
> we've been doing some tests with openstack kilo and found
> out a problem: iptables routes are not being injected to the
> router namespace.
> 
> Scenario:
> - a private network NOT connected to the outside world.
> - a router with only one interface connected to the private network.
> - a vm instance connected to the private network as well.
> 
> From inside the instance, we try to get some information from the
> metadata service with curl:
> 
> $ curl http://169.254.169.254
> curl: (7) couldn't connect to host
> 
> With the same set up in juno, there was no such problem and
> metadata information is shown.
> 
> The request is not filtered at the instance and hits the router
> namespace (checked with tcpdump). However, when looking
> from the controller at the iptables rules at the router, they appear
> empty.
> 
> stack@devstack: ~$ sudo ip netns exec
> qrouter-d4ec737a-c5fb-4f5b-8bd0-1b5353bbade3 iptables-save


> # Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015
> *filter
> :INPUT ACCEPT [5:914]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [10:868]
> COMMIT

Are you sure the l3-agent is running?  You should have seen wrapped rules from
it in most of these tables, for example:

# Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015
*filter
:INPUT ACCEPT [34:10882]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1:84]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
[...]

I would check the log files for any errors.

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Networks are not cleaned up in build failure

[Brian Haley]

On 01/15/2015 12:55 PM, Andrew Laski wrote:
> On 01/15/2015 09:33 AM, Brian Haley wrote:
>> On 01/14/2015 02:15 PM, Andrew Laski wrote:
>>> On 01/14/2015 12:57 PM, Murray, Paul (HP Cloud) wrote:
>>>> Hi All,
>>>>
>>>> I recently experienced failures getting images from Glance while spawning
>>>> instances. This step comes after building the networks in the guild 
>>>> sequence.
>>>> When the Glance failure occurred the instance was cleaned up and 
>>>> rescheduled
>>>> as expected, but the networks were not cleaned up. On investigation I found
>>>> that the cleanup code for the networks is in the compute manager’s
>>>> _/do_build_run/_instance() method as follows:
>>>>
>>>>  # NOTE(comstud): Deallocate networks if the driver wants
>>>>  # us to do so.
>>>>  if self.driver.deallocate_networks_on_reschedule(instance):
>>>>  self._cleanup_allocated_networks(context, instance,
>>>>  requested_networks)
>>>>
>>>> The default behavior in for the deallocate_networks_on_schedule() method
>>>> defined in ComputeDriver is:
>>>>
>>>>  def deallocate_networks_on_reschedule(self, instance):
>>>>  """Does the driver want networks deallocated on reschedule?"""
>>>>  return False
>>>>
>>>> Only the Ironic driver over rides this method to return True, so I think 
>>>> this
>>>> means the networks will not be cleaned up for any other virt driver.
>>>>
>>>>  
>>>> Is this really the desired behavior?
>>>>
>>> Yes.  Other than when using Ironic there is nothing specific to a particular
>>> host in the networking setup.  This means it is not necessary to deallocate 
>>> and
>>> reallocate networks when an instance is rescheduled, so we can avoid the
>>> unnecessary work of doing it.
>> That's either not true any more, or not true when DVR is enabled in Neutron,
>> since in this case the port['binding:host_id'] value has been initialized to 
>> a
>> compute node, and won't get updated when nova-conductor re-schedules the VM
>> elsewhere.
>>
>> This causes the neutron port to stay on the original compute node, and any
>> neutron operations (like floatingip-associate) happen on the "old" port, 
>> leaving
>> the VM unreachable.
> 
> Gotcha.  Then we should be rebinding that port on a reschedule or go back to
> de/reallocating.  I'm assuming there's some way to handle the port being moved
> or resizes would be broken for the same reason.
> 
> If we do need to move back to de/reallocation of networks I think it would be
> better to remove the conditional nature of it and just do it.  If the
> deallocate_networks_on_reschedule method defaults to True I don't see a case
> where it would be overridden by a driver given the information above.

Andrew,

I was able to run a test here on a multi-node setup with DVR enabled:

- Booted VM
- Associated floating IP
- Updated binding:host_id (as admin) using the neutron API:

  $ neutron port-update $port -- --binding:host_id=novacompute5

The port was correctly moved to the other compute node and the floating IP
configured.  So that showed me the agents all did the right thing as far as I
can tell.  I know Paul was looking at the nova code to try and update just this
field, I'll check-in with him regarding that so we can get a patch up soon.

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Networks are not cleaned up in build failure

[Brian Haley]

On 01/14/2015 02:15 PM, Andrew Laski wrote:
> 
> On 01/14/2015 12:57 PM, Murray, Paul (HP Cloud) wrote:
>>
>> Hi All, 
>>
>> I recently experienced failures getting images from Glance while spawning
>> instances. This step comes after building the networks in the guild sequence.
>> When the Glance failure occurred the instance was cleaned up and rescheduled
>> as expected, but the networks were not cleaned up. On investigation I found
>> that the cleanup code for the networks is in the compute manager’s
>> _/do_build_run/_instance() method as follows:
>>
>> # NOTE(comstud): Deallocate networks if the driver wants
>> # us to do so.
>> if self.driver.deallocate_networks_on_reschedule(instance):
>> self._cleanup_allocated_networks(context, instance,
>> requested_networks)
>>
>> The default behavior in for the deallocate_networks_on_schedule() method
>> defined in ComputeDriver is:
>>
>> def deallocate_networks_on_reschedule(self, instance):
>> """Does the driver want networks deallocated on reschedule?"""
>> return False
>>
>> Only the Ironic driver over rides this method to return True, so I think this
>> means the networks will not be cleaned up for any other virt driver.
>>
>>  
>>
>> Is this really the desired behavior?
>>
> 
> Yes.  Other than when using Ironic there is nothing specific to a particular
> host in the networking setup.  This means it is not necessary to deallocate 
> and
> reallocate networks when an instance is rescheduled, so we can avoid the
> unnecessary work of doing it.

That's either not true any more, or not true when DVR is enabled in Neutron,
since in this case the port['binding:host_id'] value has been initialized to a
compute node, and won't get updated when nova-conductor re-schedules the VM
elsewhere.

This causes the neutron port to stay on the original compute node, and any
neutron operations (like floatingip-associate) happen on the "old" port, leaving
the VM unreachable.

> If the instance goes to ERROR then the network will get cleaned up when the
> instance is deleted.

I think we need to clean-up even in this case now too.

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev