Public bug reported: ########## Openstack Newton OSA 14.2.4 neutron-server 9.3.2.dev3 OVS firewall_driver = openvswitch ##########
After applying a QOS DSCP-marking policy on a neutron port, the OVS flow-table on the hosting compute node does not get properly updated with the required flow to add the marking. The work-around has been to hard stop the instance, wait until the flows are removed, and re-start the instance allowing the OVS agent to rebuild the necessary flows. After the flows are fully rebuilt, the flow rule that marks traffic can be seen. neutron qos-policy-list +--------------------------------------+------------+ | id | name | +--------------------------------------+------------+ | b7c91afa-c1d1-436a-8543-e64f379d2a4f | dscp-green | | e86ab2c3-3193-40ce-8301-184be922ee6f | dscp-blue | +--------------------------------------+------------+ neutron qos-policy-show b7c91afa-c1d1-436a-8543-e64f379d2a4f +-----------------+-----------------------------------------------------------+ | Field | Value | +-----------------+-----------------------------------------------------------+ | created_at | 2017-11-21T19:23:28Z | | description | Green zone | | id | b7c91afa-c1d1-436a-8543-e64f379d2a4f | | name | dscp-green | | project_id | abcdefghilklmnop8368966eb510e105 | | revision_number | 2 | | rules | 73bb97ef-33d4-4d9e-934a-e016443648ef (type: dscp_marking) | | shared | True | | tenant_id | abcdefghilklmnop8368966eb510e105 | | updated_at | 2017-11-21T19:23:31Z | +-----------------+-----------------------------------------------------------+ neutron qos-dscp-marking-rule-show 73bb97ef-33d4-4d9e-934a-e016443648ef b7c91afa-c1d1-436a-8543-e64f379d2a4f +-----------+--------------------------------------+ | Field | Value | +-----------+--------------------------------------+ | dscp_mark | 16 | | id | 73bb97ef-33d4-4d9e-934a-e016443648ef | +-----------+--------------------------------------+ ######################## Neutron port info, *prior* to any QOS policy being applied: neutron port-show 06c15156-1cd1-4eee-b9a1-bcf379556c99 +-----------------------+----------------------------------------------------------------------------------+ | Field | Value | +-----------------------+----------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | oscomp-ho-c200 | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": false} | | binding:vif_type | ovs | | binding:vnic_type | normal | | created_at | 2017-12-06T16:50:09Z | | description | | | device_id | 50f90ac8-2e3b-43ee-a1fe-4728fb452382 | | device_owner | compute:nova | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "3767c511-f2d2-4dc3-a222-123456791011", "ip_address": "10.0.3.10"} | | id | 06c15156-1cd1-4eee-b9a1-bcf379556c99 | | mac_address | fa:16:3e:06:b1:8f | | name | | | network_id | 55555555-9c52-4658-9ca3-d3715ef54ea6 | | port_security_enabled | True | | project_id | 35aac3ee14bd447a8782871ed1cee940 | | qos_policy_id | | | revision_number | 9 | | security_groups | 26711be4-7ae8-4fbb-b097-2405bb2e4f39 | | status | ACTIVE | | tenant_id | 35aac3ee14bd447a8782871ed1cee940 | | updated_at | 2017-12-06T16:50:19Z | +-----------------------+----------------------------------------------------------------------------------+ Partial flow table off the compute linked to port/instance ... cookie=0xbfa47c9e78d2597c, duration=208.710s, table=0, n_packets=102, n_bytes=10468, idle_age=3, priority=100,in_port=8 actions=load:0x8->NXM_NX_REG5[],load:0x7->NXM_NX_REG6[],resubmit(,71) cookie=0xbfa47c9e78d2597c, duration=208.708s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=130 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=208.707s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=134 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=208.707s, table=71, n_packets=1, n_bytes=78, idle_age=205, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=135 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=208.706s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=136 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=208.706s, table=71, n_packets=11, n_bytes=462, idle_age=40, priority=95,arp,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,arp_spa=10.0.3.10 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=208.705s, table=71, n_packets=2, n_bytes=698, idle_age=205, priority=80,udp,reg5=0x8,in_port=8,tp_src=68,tp_dst=67 actions=resubmit(,73) cookie=0xbfa47c9e78d2597c, duration=208.704s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=80,udp6,reg5=0x8,in_port=8,tp_src=546,tp_dst=547 actions=resubmit(,73) cookie=0xbfa47c9e78d2597c, duration=208.703s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=70,udp,reg5=0x8,in_port=8,tp_src=67,tp_dst=68 actions=drop cookie=0xbfa47c9e78d2597c, duration=208.703s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=70,udp6,reg5=0x8,in_port=8,tp_src=547,tp_dst=546 actions=drop cookie=0xbfa47c9e78d2597c, duration=208.706s, table=71, n_packets=83, n_bytes=8840, idle_age=3, priority=65,ct_state=-trk,ip,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,nw_src=10.0.3.10 actions=ct(table=72,zone=NXM_NX_REG6[0..15]) cookie=0xbfa47c9e78d2597c, duration=208.705s, table=71, n_packets=4, n_bytes=300, idle_age=196, priority=65,ct_state=-trk,ipv6,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,ipv6_src=fe80::f816:3eff:fe06:b18f actions=ct(table=72,zone=NXM_NX_REG6[0..15]) cookie=0xbfa47c9e78d2597c, duration=208.702s, table=71, n_packets=1, n_bytes=90, idle_age=205, priority=10,ct_state=-trk,reg5=0x8,in_port=8 actions=drop ... TCPDump of the physical interface for outgoing traffic to 8.8.8.8 to view any markings: tcpdump -i bond1 -n -nn -v host 8.8.8.8 16:55:48.913100 IP (tos 0x0, ttl 63, id 39606, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.3.10 > 8.8.8.8: ICMP echo request, id 21505, seq 0, length 64 (note the tos 0x0 stating that there is no DSCP mark applied. This is expected) ######################## I then updated the port with the DSCP marking policy neutron port-update --qos-policy b7c91afa-c1d1-436a-8543-e64f379d2a4f 06c15156-1cd1-4eee-b9a1-bcf379556c99 +-----------------------+----------------------------------------------------------------------------------+ | Field | Value | +-----------------------+----------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | oscomp-ho-c200 | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": false} | | binding:vif_type | ovs | | binding:vnic_type | normal | | created_at | 2017-12-06T16:50:09Z | | description | | | device_id | 50f90ac8-2e3b-43ee-a1fe-4728fb452382 | | device_owner | compute:nova | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "3767c511-f2d2-4dc3-a222-123456791011", "ip_address": "10.0.3.10"} | | id | 06c15156-1cd1-4eee-b9a1-bcf379556c99 | | mac_address | fa:16:3e:06:b1:8f | | name | | | network_id | 55555555-9c52-4658-9ca3-d3715ef54ea6 | | port_security_enabled | True | | project_id | 35aac3ee14bd447a8782871ed1cee940 | | qos_policy_id | b7c91afa-c1d1-436a-8543-e64f379d2a4f | | revision_number | 12 | | security_groups | 26711be4-7ae8-4fbb-b097-2405bb2e4f39 | | status | ACTIVE | | tenant_id | 35aac3ee14bd447a8782871ed1cee940 | | updated_at | 2017-12-06T16:58:03Z | +-----------------------+----------------------------------------------------------------------------------+ (The qos policy can been seen applied to the port) OVS agent log files on the compute for the port-update: 2017-12-06 16:58:02.910 21677 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-8a8d3edc-f0cd-450f-b77b-d46729fc7bb8 - - - - -] Port 06c15156-1cd1-4eee-b9a1-bcf379556c99 updated. Details: {u'profile': {}, u'network_qos_policy_id': None, u'qos_policy_id': u'b7c91afa-c1d1-436a-8543-e64f379d2a4f', u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'55555555-9c52-4658-9ca3-d3715ef54ea6', u'segmentation_id': 2007, u'device_owner': u'compute:nova', u'physical_network': u'physnet1', u'mac_address': u'fa:16:3e:06:b1:8f', u'device': u'06c15156-1cd1-4eee-b9a1-bcf379556c99', u'port_security_enabled': True, u'port_id': u'06c15156-1cd1-4eee-b9a1-bcf379556c99', u'fixed_ips': [{u'subnet_id': u'3767c511-f2d2-4dc3-a222-123456791011', u'ip_address': u'10.0.3.10'}], u'network_type': u'vlan'} 2017-12-06 16:58:09.322 21677 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-8a8d3edc-f0cd-450f-b77b-d46729fc7bb8 - - - - -] Configuration for devices up [u'06c15156-1cd1-4eee-b9a1-bcf379556c99'] and devices down [] completed. After a few minutes of waiting, there are still no flows to modify traffic with the DSCP mark: OVS FLOWS ... cookie=0xbfa47c9e78d2597c, duration=60.075s, table=0, n_packets=13, n_bytes=878, idle_age=1, priority=100,in_port=8 actions=load:0x8->NXM_NX_REG5[],load:0x7->NXM_NX_REG6[],resubmit(,71) cookie=0xbfa47c9e78d2597c, duration=60.073s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=130 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=60.072s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=134 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=60.072s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=135 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=60.071s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=136 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=60.071s, table=71, n_packets=2, n_bytes=84, idle_age=23, priority=95,arp,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,arp_spa=10.0.3.10 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=60.070s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=80,udp,reg5=0x8,in_port=8,tp_src=68,tp_dst=67 actions=resubmit(,73) cookie=0xbfa47c9e78d2597c, duration=60.069s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=80,udp6,reg5=0x8,in_port=8,tp_src=546,tp_dst=547 actions=resubmit(,73) cookie=0xbfa47c9e78d2597c, duration=60.069s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=70,udp,reg5=0x8,in_port=8,tp_src=67,tp_dst=68 actions=drop cookie=0xbfa47c9e78d2597c, duration=60.068s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=70,udp6,reg5=0x8,in_port=8,tp_src=547,tp_dst=546 actions=drop cookie=0xbfa47c9e78d2597c, duration=60.071s, table=71, n_packets=11, n_bytes=794, idle_age=1, priority=65,ct_state=-trk,ip,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,nw_src=10.0.3.10 actions=ct(table=72,zone=NXM_NX_REG6[0..15]) cookie=0xbfa47c9e78d2597c, duration=60.070s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=65,ct_state=-trk,ipv6,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,ipv6_src=fe80::f816:3eff:fe06:b18f actions=ct(table=72,zone=NXM_NX_REG6[0..15]) cookie=0xbfa47c9e78d2597c, duration=60.068s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=10,ct_state=-trk,reg5=0x8,in_port=8 actions=drop ... TCPDump still shows no tos mark: TCPDUMP tcpdump -i bond1 -n -nn -v host 8.8.8.8 17:00:37.167559 IP (tos 0x0, ttl 63, id 38836, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.3.10 > 8.8.8.8: ICMP echo request, id 21761, seq 0, length 64 ######################## As a workaround, the instance was hard stopped allowing the flows to be deleted. Then the instance is started and the flows re-created. nova stop instance_id (wait a few moments) nova start instance_id Here is a new snipit of the flow-table with the "mod_nw_tos" action. (first line below). OVS FLOWS ... cookie=0xb6082f15d4334178, duration=447.524s, table=0, n_packets=30, n_bytes=3864, idle_age=3, priority=65535,reg2=0,in_port=10 actions=mod_nw_tos:64,load:0x37->NXM_NX_REG2[0..5],resubmit(,0) cookie=0xbfa47c9e78d2597c, duration=447.234s, table=0, n_packets=30, n_bytes=3864, idle_age=3, priority=100,in_port=10 actions=load:0xa->NXM_NX_REG5[],load:0x7->NXM_NX_REG6[],resubmit(,71) cookie=0xbfa47c9e78d2597c, duration=447.232s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=130 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=447.232s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=134 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=447.231s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=135 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=136 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=5, n_bytes=210, idle_age=15, priority=95,arp,reg5=0xa,in_port=10,dl_src=fa:16:3e:06:b1:8f,arp_spa=10.0.3.10 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=447.229s, table=71, n_packets=2, n_bytes=698, idle_age=390, priority=80,udp,reg5=0xa,in_port=10,tp_src=68,tp_dst=67 actions=resubmit(,73) cookie=0xbfa47c9e78d2597c, duration=447.228s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=80,udp6,reg5=0xa,in_port=10,tp_src=546,tp_dst=547 actions=resubmit(,73) cookie=0xbfa47c9e78d2597c, duration=447.228s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=70,udp,reg5=0xa,in_port=10,tp_src=67,tp_dst=68 actions=drop cookie=0xbfa47c9e78d2597c, duration=447.227s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=70,udp6,reg5=0xa,in_port=10,tp_src=547,tp_dst=546 actions=drop cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=20, n_bytes=2726, idle_age=3, priority=65,ct_state=-trk,ip,reg5=0xa,in_port=10,dl_src=fa:16:3e:06:b1:8f,nw_src=10.0.3.10 actions=ct(table=72,zone=NXM_NX_REG6[0..15 ]) cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=3, n_bytes=230, idle_age=441, priority=65,ct_state=-trk,ipv6,reg5=0xa,in_port=10,dl_src=fa:16:3e:06:b1:8f,ipv6_src=fe80::f816:3eff:fe06:b18f actions=ct(table=72,zo ne=NXM_NX_REG6[0..15]) cookie=0xbfa47c9e78d2597c, duration=447.227s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=10,ct_state=-trk,reg5=0xa,in_port=10 actions=drop ... TCPDUMP: tcpdump -i bond1 -n -nn -v host 8.8.8.8 17:13:37.694875 IP (tos 0x40, ttl 63, id 32155, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.3.10 > 8.8.8.8: ICMP echo request, id 12801, seq 0, length 64 TCPDump show tos 0x40 ######################## Interestingly, we do not see the same behavior on OVS environments using the hybrid firewall driver. We only see it when using the OVS firewall driver. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1736792 Title: DSCP marking QOS policy applied to port not properly updating OVS flow table Status in neutron: New Bug description: ########## Openstack Newton OSA 14.2.4 neutron-server 9.3.2.dev3 OVS firewall_driver = openvswitch ########## After applying a QOS DSCP-marking policy on a neutron port, the OVS flow-table on the hosting compute node does not get properly updated with the required flow to add the marking. The work-around has been to hard stop the instance, wait until the flows are removed, and re- start the instance allowing the OVS agent to rebuild the necessary flows. After the flows are fully rebuilt, the flow rule that marks traffic can be seen. neutron qos-policy-list +--------------------------------------+------------+ | id | name | +--------------------------------------+------------+ | b7c91afa-c1d1-436a-8543-e64f379d2a4f | dscp-green | | e86ab2c3-3193-40ce-8301-184be922ee6f | dscp-blue | +--------------------------------------+------------+ neutron qos-policy-show b7c91afa-c1d1-436a-8543-e64f379d2a4f +-----------------+-----------------------------------------------------------+ | Field | Value | +-----------------+-----------------------------------------------------------+ | created_at | 2017-11-21T19:23:28Z | | description | Green zone | | id | b7c91afa-c1d1-436a-8543-e64f379d2a4f | | name | dscp-green | | project_id | abcdefghilklmnop8368966eb510e105 | | revision_number | 2 | | rules | 73bb97ef-33d4-4d9e-934a-e016443648ef (type: dscp_marking) | | shared | True | | tenant_id | abcdefghilklmnop8368966eb510e105 | | updated_at | 2017-11-21T19:23:31Z | +-----------------+-----------------------------------------------------------+ neutron qos-dscp-marking-rule-show 73bb97ef-33d4-4d9e-934a-e016443648ef b7c91afa-c1d1-436a-8543-e64f379d2a4f +-----------+--------------------------------------+ | Field | Value | +-----------+--------------------------------------+ | dscp_mark | 16 | | id | 73bb97ef-33d4-4d9e-934a-e016443648ef | +-----------+--------------------------------------+ ######################## Neutron port info, *prior* to any QOS policy being applied: neutron port-show 06c15156-1cd1-4eee-b9a1-bcf379556c99 +-----------------------+----------------------------------------------------------------------------------+ | Field | Value | +-----------------------+----------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | oscomp-ho-c200 | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": false} | | binding:vif_type | ovs | | binding:vnic_type | normal | | created_at | 2017-12-06T16:50:09Z | | description | | | device_id | 50f90ac8-2e3b-43ee-a1fe-4728fb452382 | | device_owner | compute:nova | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "3767c511-f2d2-4dc3-a222-123456791011", "ip_address": "10.0.3.10"} | | id | 06c15156-1cd1-4eee-b9a1-bcf379556c99 | | mac_address | fa:16:3e:06:b1:8f | | name | | | network_id | 55555555-9c52-4658-9ca3-d3715ef54ea6 | | port_security_enabled | True | | project_id | 35aac3ee14bd447a8782871ed1cee940 | | qos_policy_id | | | revision_number | 9 | | security_groups | 26711be4-7ae8-4fbb-b097-2405bb2e4f39 | | status | ACTIVE | | tenant_id | 35aac3ee14bd447a8782871ed1cee940 | | updated_at | 2017-12-06T16:50:19Z | +-----------------------+----------------------------------------------------------------------------------+ Partial flow table off the compute linked to port/instance ... cookie=0xbfa47c9e78d2597c, duration=208.710s, table=0, n_packets=102, n_bytes=10468, idle_age=3, priority=100,in_port=8 actions=load:0x8->NXM_NX_REG5[],load:0x7->NXM_NX_REG6[],resubmit(,71) cookie=0xbfa47c9e78d2597c, duration=208.708s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=130 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=208.707s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=134 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=208.707s, table=71, n_packets=1, n_bytes=78, idle_age=205, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=135 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=208.706s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=136 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=208.706s, table=71, n_packets=11, n_bytes=462, idle_age=40, priority=95,arp,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,arp_spa=10.0.3.10 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=208.705s, table=71, n_packets=2, n_bytes=698, idle_age=205, priority=80,udp,reg5=0x8,in_port=8,tp_src=68,tp_dst=67 actions=resubmit(,73) cookie=0xbfa47c9e78d2597c, duration=208.704s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=80,udp6,reg5=0x8,in_port=8,tp_src=546,tp_dst=547 actions=resubmit(,73) cookie=0xbfa47c9e78d2597c, duration=208.703s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=70,udp,reg5=0x8,in_port=8,tp_src=67,tp_dst=68 actions=drop cookie=0xbfa47c9e78d2597c, duration=208.703s, table=71, n_packets=0, n_bytes=0, idle_age=208, priority=70,udp6,reg5=0x8,in_port=8,tp_src=547,tp_dst=546 actions=drop cookie=0xbfa47c9e78d2597c, duration=208.706s, table=71, n_packets=83, n_bytes=8840, idle_age=3, priority=65,ct_state=-trk,ip,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,nw_src=10.0.3.10 actions=ct(table=72,zone=NXM_NX_REG6[0..15]) cookie=0xbfa47c9e78d2597c, duration=208.705s, table=71, n_packets=4, n_bytes=300, idle_age=196, priority=65,ct_state=-trk,ipv6,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,ipv6_src=fe80::f816:3eff:fe06:b18f actions=ct(table=72,zone=NXM_NX_REG6[0..15]) cookie=0xbfa47c9e78d2597c, duration=208.702s, table=71, n_packets=1, n_bytes=90, idle_age=205, priority=10,ct_state=-trk,reg5=0x8,in_port=8 actions=drop ... TCPDump of the physical interface for outgoing traffic to 8.8.8.8 to view any markings: tcpdump -i bond1 -n -nn -v host 8.8.8.8 16:55:48.913100 IP (tos 0x0, ttl 63, id 39606, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.3.10 > 8.8.8.8: ICMP echo request, id 21505, seq 0, length 64 (note the tos 0x0 stating that there is no DSCP mark applied. This is expected) ######################## I then updated the port with the DSCP marking policy neutron port-update --qos-policy b7c91afa-c1d1-436a-8543-e64f379d2a4f 06c15156-1cd1-4eee-b9a1-bcf379556c99 +-----------------------+----------------------------------------------------------------------------------+ | Field | Value | +-----------------------+----------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | oscomp-ho-c200 | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": false} | | binding:vif_type | ovs | | binding:vnic_type | normal | | created_at | 2017-12-06T16:50:09Z | | description | | | device_id | 50f90ac8-2e3b-43ee-a1fe-4728fb452382 | | device_owner | compute:nova | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "3767c511-f2d2-4dc3-a222-123456791011", "ip_address": "10.0.3.10"} | | id | 06c15156-1cd1-4eee-b9a1-bcf379556c99 | | mac_address | fa:16:3e:06:b1:8f | | name | | | network_id | 55555555-9c52-4658-9ca3-d3715ef54ea6 | | port_security_enabled | True | | project_id | 35aac3ee14bd447a8782871ed1cee940 | | qos_policy_id | b7c91afa-c1d1-436a-8543-e64f379d2a4f | | revision_number | 12 | | security_groups | 26711be4-7ae8-4fbb-b097-2405bb2e4f39 | | status | ACTIVE | | tenant_id | 35aac3ee14bd447a8782871ed1cee940 | | updated_at | 2017-12-06T16:58:03Z | +-----------------------+----------------------------------------------------------------------------------+ (The qos policy can been seen applied to the port) OVS agent log files on the compute for the port-update: 2017-12-06 16:58:02.910 21677 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-8a8d3edc-f0cd-450f-b77b-d46729fc7bb8 - - - - -] Port 06c15156-1cd1-4eee-b9a1-bcf379556c99 updated. Details: {u'profile': {}, u'network_qos_policy_id': None, u'qos_policy_id': u'b7c91afa-c1d1-436a-8543-e64f379d2a4f', u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'55555555-9c52-4658-9ca3-d3715ef54ea6', u'segmentation_id': 2007, u'device_owner': u'compute:nova', u'physical_network': u'physnet1', u'mac_address': u'fa:16:3e:06:b1:8f', u'device': u'06c15156-1cd1-4eee-b9a1-bcf379556c99', u'port_security_enabled': True, u'port_id': u'06c15156-1cd1-4eee-b9a1-bcf379556c99', u'fixed_ips': [{u'subnet_id': u'3767c511-f2d2-4dc3-a222-123456791011', u'ip_address': u'10.0.3.10'}], u'network_type': u'vlan'} 2017-12-06 16:58:09.322 21677 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-8a8d3edc-f0cd-450f-b77b-d46729fc7bb8 - - - - -] Configuration for devices up [u'06c15156-1cd1-4eee-b9a1-bcf379556c99'] and devices down [] completed. After a few minutes of waiting, there are still no flows to modify traffic with the DSCP mark: OVS FLOWS ... cookie=0xbfa47c9e78d2597c, duration=60.075s, table=0, n_packets=13, n_bytes=878, idle_age=1, priority=100,in_port=8 actions=load:0x8->NXM_NX_REG5[],load:0x7->NXM_NX_REG6[],resubmit(,71) cookie=0xbfa47c9e78d2597c, duration=60.073s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=130 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=60.072s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=134 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=60.072s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=135 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=60.071s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=95,icmp6,reg5=0x8,in_port=8,icmp_type=136 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=60.071s, table=71, n_packets=2, n_bytes=84, idle_age=23, priority=95,arp,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,arp_spa=10.0.3.10 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=60.070s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=80,udp,reg5=0x8,in_port=8,tp_src=68,tp_dst=67 actions=resubmit(,73) cookie=0xbfa47c9e78d2597c, duration=60.069s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=80,udp6,reg5=0x8,in_port=8,tp_src=546,tp_dst=547 actions=resubmit(,73) cookie=0xbfa47c9e78d2597c, duration=60.069s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=70,udp,reg5=0x8,in_port=8,tp_src=67,tp_dst=68 actions=drop cookie=0xbfa47c9e78d2597c, duration=60.068s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=70,udp6,reg5=0x8,in_port=8,tp_src=547,tp_dst=546 actions=drop cookie=0xbfa47c9e78d2597c, duration=60.071s, table=71, n_packets=11, n_bytes=794, idle_age=1, priority=65,ct_state=-trk,ip,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,nw_src=10.0.3.10 actions=ct(table=72,zone=NXM_NX_REG6[0..15]) cookie=0xbfa47c9e78d2597c, duration=60.070s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=65,ct_state=-trk,ipv6,reg5=0x8,in_port=8,dl_src=fa:16:3e:06:b1:8f,ipv6_src=fe80::f816:3eff:fe06:b18f actions=ct(table=72,zone=NXM_NX_REG6[0..15]) cookie=0xbfa47c9e78d2597c, duration=60.068s, table=71, n_packets=0, n_bytes=0, idle_age=60, priority=10,ct_state=-trk,reg5=0x8,in_port=8 actions=drop ... TCPDump still shows no tos mark: TCPDUMP tcpdump -i bond1 -n -nn -v host 8.8.8.8 17:00:37.167559 IP (tos 0x0, ttl 63, id 38836, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.3.10 > 8.8.8.8: ICMP echo request, id 21761, seq 0, length 64 ######################## As a workaround, the instance was hard stopped allowing the flows to be deleted. Then the instance is started and the flows re-created. nova stop instance_id (wait a few moments) nova start instance_id Here is a new snipit of the flow-table with the "mod_nw_tos" action. (first line below). OVS FLOWS ... cookie=0xb6082f15d4334178, duration=447.524s, table=0, n_packets=30, n_bytes=3864, idle_age=3, priority=65535,reg2=0,in_port=10 actions=mod_nw_tos:64,load:0x37->NXM_NX_REG2[0..5],resubmit(,0) cookie=0xbfa47c9e78d2597c, duration=447.234s, table=0, n_packets=30, n_bytes=3864, idle_age=3, priority=100,in_port=10 actions=load:0xa->NXM_NX_REG5[],load:0x7->NXM_NX_REG6[],resubmit(,71) cookie=0xbfa47c9e78d2597c, duration=447.232s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=130 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=447.232s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=134 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=447.231s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=135 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=95,icmp6,reg5=0xa,in_port=10,icmp_type=136 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=5, n_bytes=210, idle_age=15, priority=95,arp,reg5=0xa,in_port=10,dl_src=fa:16:3e:06:b1:8f,arp_spa=10.0.3.10 actions=NORMAL cookie=0xbfa47c9e78d2597c, duration=447.229s, table=71, n_packets=2, n_bytes=698, idle_age=390, priority=80,udp,reg5=0xa,in_port=10,tp_src=68,tp_dst=67 actions=resubmit(,73) cookie=0xbfa47c9e78d2597c, duration=447.228s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=80,udp6,reg5=0xa,in_port=10,tp_src=546,tp_dst=547 actions=resubmit(,73) cookie=0xbfa47c9e78d2597c, duration=447.228s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=70,udp,reg5=0xa,in_port=10,tp_src=67,tp_dst=68 actions=drop cookie=0xbfa47c9e78d2597c, duration=447.227s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=70,udp6,reg5=0xa,in_port=10,tp_src=547,tp_dst=546 actions=drop cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=20, n_bytes=2726, idle_age=3, priority=65,ct_state=-trk,ip,reg5=0xa,in_port=10,dl_src=fa:16:3e:06:b1:8f,nw_src=10.0.3.10 actions=ct(table=72,zone=NXM_NX_REG6[0..15 ]) cookie=0xbfa47c9e78d2597c, duration=447.230s, table=71, n_packets=3, n_bytes=230, idle_age=441, priority=65,ct_state=-trk,ipv6,reg5=0xa,in_port=10,dl_src=fa:16:3e:06:b1:8f,ipv6_src=fe80::f816:3eff:fe06:b18f actions=ct(table=72,zo ne=NXM_NX_REG6[0..15]) cookie=0xbfa47c9e78d2597c, duration=447.227s, table=71, n_packets=0, n_bytes=0, idle_age=447, priority=10,ct_state=-trk,reg5=0xa,in_port=10 actions=drop ... TCPDUMP: tcpdump -i bond1 -n -nn -v host 8.8.8.8 17:13:37.694875 IP (tos 0x40, ttl 63, id 32155, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.3.10 > 8.8.8.8: ICMP echo request, id 12801, seq 0, length 64 TCPDump show tos 0x40 ######################## Interestingly, we do not see the same behavior on OVS environments using the hybrid firewall driver. We only see it when using the OVS firewall driver. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1736792/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp