Hi, I have been checking CSIT failures in 3-node mitaka/newton jobs. I noticed that after reboot scenario of 2 ODLs some flows containing nicira extensions are being rewritten without the containing the nicira actions.
Examples from latest reports: 1. ARP reply flow in GW_MAC table Before ODL1+ODL2 startup cookie=0x1080000, duration=3104.003s, table=19, n_packets=40, n_bytes=1680, priority=100,arp,arp_op=2 actions=CONTROLLER:65535,resubmit(,17) https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-3node-openstack-newton-upstream-learn-boron/55/robot/report/log.html#s1-s3-s2-t27-k5-k1-k1-k1-k16-k3 After ODL1+ODL2 startup - resubmit is missing cookie=0x1080000, duration=0.227s, table=19, n_packets=40, n_bytes=1680, priority=100,arp,arp_op=2 actions=CONTROLLER:65535 https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-3node-openstack-newton-upstream-learn-boron/55/robot/report/log.html#s1-s3-s2-t28-k3-k1-k1-k1-k16 2. ARP responder flow for router interface in ARP_RESPONDER table Before ODL1+ODL2 startup cookie=0x122c3d7e, duration=8603.194s, table=81, n_packets=1, n_bytes=42, priority=100,arp,metadata=0x70337900000222e6/0xffffff00fffffffe,arp_tpa=10.10.10.4,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:0d:c3:35->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0dc335->NXM_NX_ARP_SHA[],load:0xa0a0a04->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x337900->NXM_NX_REG6[],write_metadata:0/0x1,goto_table:220 cookie=0x62220003, duration=3182.514s, table=81, n_packets=2, n_bytes=84, priority=100,arp,metadata=0x70000100000251c2/0xffffff00fffffffe,arp_tpa=90.0.0.1,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:c7:33:48->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163ec73348->NXM_NX_ARP_SHA[],load:0x5a000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x100->NXM_NX_REG6[],resubmit(,220) cookie=0x6c22000a, duration=3177.981s, table=81, n_packets=4, n_bytes=168, priority=100,arp,metadata=0x70000800000251c2/0xffffff00fffffffe,arp_tpa=100.0.0.1,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:7f:b9:6d->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e7fb96d->NXM_NX_ARP_SHA[],load:0x64000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x800->NXM_NX_REG6[],resubmit(,220) cookie=0x6c220005, duration=3177.909s, table=81, n_packets=4, n_bytes=168, priority=100,arp,metadata=0x70000300000251c2/0xffffff00fffffffe,arp_tpa=100.0.0.1,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:7f:b9:6d->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e7fb96d->NXM_NX_ARP_SHA[],load:0x64000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x300->NXM_NX_REG6[],resubmit(,220) https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-3node-openstack-mitaka-upstream-learn-carbon/60/robot/report/log.html#s1-s3-s2-t27-k5-k1-k2-k1-k16-k3 After ODL1+ODL2 startup - flows for arp_tpa 90.0.0.1/100.0.0.1 all nicira actions are missing cookie=0x122c3d7e, duration=9636.373s, table=81, n_packets=1, n_bytes=42, priority=100,arp,metadata=0x70337900000222e6/0xffffff00fffffffe,arp_tpa=10.10.10.4,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:0d:c3:35->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0dc335->NXM_NX_ARP_SHA[],load:0xa0a0a04->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x337900->NXM_NX_REG6[],write_metadata:0/0x1,goto_table:220 cookie=0x6c22000a, duration=4211.160s, table=81, n_packets=4, n_bytes=168, priority=100,arp,metadata=0x70000800000251c2/0xffffff00fffffffe,arp_tpa=100.0.0.1,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:3e:7f:b9:6d->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e7fb96d->NXM_NX_ARP_SHA[],load:0x64000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x800->NXM_NX_REG6[],resubmit(,220) cookie=0x62220003, duration=7.947s, table=81, n_packets=2, n_bytes=84, priority=100,arp,metadata=0x70000100000251c2/0xffffff00fffffffe,arp_tpa=90.0.0.1,arp_op=1 actions=set_field:fa:16:3e:c7:33:48->eth_src cookie=0x6c220005, duration=7.947s, table=81, n_packets=4, n_bytes=168, priority=100,arp,metadata=0x70000300000251c2/0xffffff00fffffffe,arp_tpa=100.0.0.1,arp_op=1 actions=set_field:fa:16:3e:7f:b9:6d->eth_src https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-3node-openstack-mitaka-upstream-learn-carbon/60/robot/report/log.html#s1-s3-s2-t28-k3-k1-k2-k1-k16-k3 In the karaf log of ODL3 (that hasn't been restarted) there are lots of these errors at the time flows get corrupted 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor | 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to process interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase for version 4 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor | 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to process interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegMoveRpcUpdateFlowUpdatedApplyActionsCase for version 4 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor | 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to process interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegMoveRpcUpdateFlowUpdatedApplyActionsCase for version 4 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor | 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to process interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase for version 4 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor | 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to process interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase for version 4 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor | 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to process interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase for version 4 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor | 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to process interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionRegLoadRpcUpdateFlowUpdatedApplyActionsCase for version 4 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | ConvertorProcessor | 307 - org.opendaylight.openflowplugin - 0.4.0.SNAPSHOT | Failed to process interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.action.rev140714.update.flow.input.updated.flow.instructions.instruction.instruction.apply.actions._case.apply.actions.action.action.NxActionResubmitRpcUpdateFlowUpdatedApplyActionsCase for version 4 https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-newton-upstream-learn-boron/55/archives/odl3_karaf.log.gz https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-mitaka-upstream-learn-carbon/60/archives/odl3_karaf.log.gz Has anyone encountered this before or have any idea how to fix it? Thanks Tali
_______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
