Hi Shuva,
These results were collected from netvirt csit 3node reports for both Carbon
and Boron.
There are two main issues we haven’t seen in the 1node setups
1. pkt_out messages containing nicira actions were sent to the wire
without the actions and the following log appears
Failed to process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.transmit.packet.input.action.action.NxActionRegLoadRpcTransmitPacketCase
for version 4
Failed to process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.transmit.packet.input.action.action.NxActionResubmitRpcTransmitPacketCase
for version 4
After adding the missing TransmitPacket cases to NiciraExtensionProvider [1]
the problem is resolved.
2. During ODL reboot scenarios some flows were rewritten without nicira
actions and the following log appears
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegMoveRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegMoveRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionResubmitRpcUpdateFlowUpdatedApplyActionsCase
for version 4
This problem was also resolved after adding the missing cases to
NiciraExtensionProvider.
But could you explain why these types of issues happen only in cluster? Is it
related to routed RPCs? The problem is also sporadic and some of the csit pass
successfully. Is this because of some RPC optimization happening where the ODL
who is the master of the switch is also the one sending the pkt_out/installing
the flows?
Also, I created patch [1] only to solve netvirt csit issues. Would it be
correct to register nicira actions for all the missing subclasses of
NxAction*Grouping to avoid more of the same?
[1] https://git.opendaylight.org/gerrit/#/c/50029
Thanks
Tali
From: Shuva Kar [mailto:[email protected]]
Sent: Wednesday, January 4, 2017 6:59 PM
To: Ben-Meir, Tali <[email protected]>
Subject: Re: [openflowplugin-dev] Flows with nicira extensions get corrupted
after cluster reboot
Hi Tali,
A few questions:
1. After ODL restart, do we see the flows going over the wireshark? If so do we
see any errors?
2. Also post restart do we see the flows in the inventoy-config Datastore?
3. I guess the ODL version used is Boron(Bo) , if not please correct me.
4. I guess we have the nicira extensions features installed post restart..
Br,shuva
On Mon, Jan 2, 2017 at 11:04 PM, Ben-Meir, Tali
<[email protected]<mailto:[email protected]>> wrote:
Hi,
I have been checking CSIT failures in 3-node mitaka/newton jobs.
I noticed that after reboot scenario of 2 ODLs some flows containing nicira
extensions are being rewritten without the containing the nicira actions.
Examples from latest reports:
1. ARP reply flow in GW_MAC table
Before ODL1+ODL2 startup
cookie=0x1080000, duration=3104.003s, table=19, n_packets=40, n_bytes=1680,
priority=100,arp,arp_op=2 actions=CONTROLLER:65535,resubmit(,17)
https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-3node-openstack-newton-upstream-learn-boron/55/robot/report/log.html#s1-s3-s2-t27-k5-k1-k1-k1-k16-k3
After ODL1+ODL2 startup – resubmit is missing
cookie=0x1080000, duration=0.227s, table=19, n_packets=40, n_bytes=1680,
priority=100,arp,arp_op=2 actions=CONTROLLER:65535
https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-3node-openstack-newton-upstream-learn-boron/55/robot/report/log.html#s1-s3-s2-t28-k3-k1-k1-k1-k16
2. ARP responder flow for router interface in ARP_RESPONDER table
Before ODL1+ODL2 startup
cookie=0x122c3d7e, duration=8603.194s, table=81, n_packets=1, n_bytes=42,
priority=100,arp,metadata=0x70337900000222e6/0xffffff00fffffffe,arp_tpa=10.10.10.4,arp_op=1
actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:0d:c3:35->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0dc335->NXM_NX_ARP_SHA[],load:0xa0a0a04->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x337900->NXM_NX_REG6[],write_metadata:0/0x1,goto_table:220
cookie=0x62220003, duration=3182.514s, table=81, n_packets=2, n_bytes=84,
priority=100,arp,metadata=0x70000100000251c2/0xffffff00fffffffe,arp_tpa=90.0.0.1,arp_op=1
actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:c7:33:48->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163ec73348->NXM_NX_ARP_SHA[],load:0x5a000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x100->NXM_NX_REG6[],resubmit(,220)
cookie=0x6c22000a, duration=3177.981s, table=81, n_packets=4, n_bytes=168,
priority=100,arp,metadata=0x70000800000251c2/0xffffff00fffffffe,arp_tpa=100.0.0.1,arp_op=1
actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:7f:b9:6d->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e7fb96d->NXM_NX_ARP_SHA[],load:0x64000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x800->NXM_NX_REG6[],resubmit(,220)
cookie=0x6c220005, duration=3177.909s, table=81, n_packets=4, n_bytes=168,
priority=100,arp,metadata=0x70000300000251c2/0xffffff00fffffffe,arp_tpa=100.0.0.1,arp_op=1
actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:7f:b9:6d->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e7fb96d->NXM_NX_ARP_SHA[],load:0x64000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x300->NXM_NX_REG6[],resubmit(,220)
https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-3node-openstack-mitaka-upstream-learn-carbon/60/robot/report/log.html#s1-s3-s2-t27-k5-k1-k2-k1-k16-k3
After ODL1+ODL2 startup – flows for arp_tpa
90.0.0.1/100.0.0.1<http://90.0.0.1/100.0.0.1> all nicira actions are missing
cookie=0x122c3d7e, duration=9636.373s, table=81, n_packets=1, n_bytes=42,
priority=100,arp,metadata=0x70337900000222e6/0xffffff00fffffffe,arp_tpa=10.10.10.4,arp_op=1
actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:0d:c3:35->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0dc335->NXM_NX_ARP_SHA[],load:0xa0a0a04->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x337900->NXM_NX_REG6[],write_metadata:0/0x1,goto_table:220
cookie=0x6c22000a, duration=4211.160s, table=81, n_packets=4, n_bytes=168,
priority=100,arp,metadata=0x70000800000251c2/0xffffff00fffffffe,arp_tpa=100.0.0.1,arp_op=1
actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:7f:b9:6d->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e7fb96d->NXM_NX_ARP_SHA[],load:0x64000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x800->NXM_NX_REG6[],resubmit(,220)
cookie=0x62220003, duration=7.947s, table=81, n_packets=2, n_bytes=84,
priority=100,arp,metadata=0x70000100000251c2/0xffffff00fffffffe,arp_tpa=90.0.0.1,arp_op=1
actions=set_field:fa:16:3e:c7:33:48->eth_src
cookie=0x6c220005, duration=7.947s, table=81, n_packets=4, n_bytes=168,
priority=100,arp,metadata=0x70000300000251c2/0xffffff00fffffffe,arp_tpa=100.0.0.1,arp_op=1
actions=set_field:fa:16:3e:7f:b9:6d->eth_src
https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-3node-openstack-mitaka-upstream-learn-carbon/60/robot/report/log.html#s1-s3-s2-t28-k3-k1-k2-k1-k16-k3
In the karaf log of ODL3 (that hasn’t been restarted) there are lots of these
errors at the time flows get corrupted
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegMoveRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegMoveRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase
for version 4
2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor
| 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to
process interface
org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionResubmitRpcUpdateFlowUpdatedApplyActionsCase
for version 4
https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-newton-upstream-learn-boron/55/archives/odl3_karaf.log.gz
https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-mitaka-upstream-learn-carbon/60/archives/odl3_karaf.log.gz
Has anyone encountered this before or have any idea how to fix it?
Thanks
Tali
_______________________________________________
openflowplugin-dev mailing list
[email protected]<mailto:[email protected]>
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
_______________________________________________
openflowplugin-dev mailing list
[email protected]
https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
