Hi Tali, The change-set in [1] looks good!.
As for your specific queries : Flow/group/meter additions in a 3 node cluster do not go through the routed rpc route . We always provision to the switches that are local to the controller - as in reference [A]. Even in the case of reconciliation, we check the switch locality as in [B] and then provision. We need to debug further to say anything more, probably checking whether the "isNodeOwner" is behaving properly could be a starting point for us. [A] preconfiguration check in AbstractListeningCommiter [B] isNodeOwner in FlowNodeReconciliationImpl Hope it helps. Br,shuva On Thu, Jan 5, 2017 at 6:48 PM, Ben-Meir, Tali <[email protected]> wrote: > Hi Shuva, > > > > These results were collected from netvirt csit 3node reports for both > Carbon and Boron. > > There are two main issues we haven’t seen in the 1node setups > > 1. pkt_out messages containing nicira actions were sent to the wire > without the actions and the following log appears > > Failed to process interface org.opendaylight.yang.gen.v1.u > rn.opendaylight.openflowplugin.extension.nicira.action. > rev140714.transmit.packet.input.action.action.NxActionRegLoadRpcTransmitPacketCase > for version 4 > > Failed to process interface org.opendaylight.yang.gen.v1.u > rn.opendaylight.openflowplugin.extension.nicira.action. > rev140714.transmit.packet.input.action.action.NxActionResubmitRpcTransmitPacketCase > for version 4 > > > > After adding the missing TransmitPacket cases to NiciraExtensionProvider > [1] the problem is resolved. > > > > 2. During ODL reboot scenarios some flows were rewritten without > nicira actions and the following log appears > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegLoadR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegMoveR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegMoveR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegLoadR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegLoadR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegLoadR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegLoadR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionResubmit > RpcUpdateFlowUpdatedApplyActionsCase for version 4 > > This problem was also resolved after adding the missing cases to > NiciraExtensionProvider. > > > > But could you explain why these types of issues happen only in cluster? Is > it related to routed RPCs? The problem is also sporadic and some of the > csit pass successfully. Is this because of some RPC optimization happening > where the ODL who is the master of the switch is also the one sending the > pkt_out/installing the flows? > > > > Also, I created patch [1] only to solve netvirt csit issues. Would it be > correct to register nicira actions for all the missing subclasses of > NxAction*Grouping to avoid more of the same? > > > > [1] https://git.opendaylight.org/gerrit/#/c/50029 > > > > > > Thanks > Tali > > > > *From:* Shuva Kar [mailto:[email protected]] > *Sent:* Wednesday, January 4, 2017 6:59 PM > *To:* Ben-Meir, Tali <[email protected]> > *Subject:* Re: [openflowplugin-dev] Flows with nicira extensions get > corrupted after cluster reboot > > > > Hi Tali, > > > > A few questions: > > > > 1. After ODL restart, do we see the flows going over the wireshark? If so > do we see any errors? > > 2. Also post restart do we see the flows in the inventoy-config Datastore? > > 3. I guess the ODL version used is Boron(Bo) , if not please correct me. > > 4. I guess we have the nicira extensions features installed post restart.. > > > > Br,shuva > > > > On Mon, Jan 2, 2017 at 11:04 PM, Ben-Meir, Tali <[email protected]> > wrote: > > Hi, > > > > I have been checking CSIT failures in 3-node mitaka/newton jobs. > > I noticed that after reboot scenario of 2 ODLs some flows containing > nicira extensions are being rewritten without the containing the nicira > actions. > > > > Examples from latest reports: > > > > 1. ARP reply flow in GW_MAC table > > Before ODL1+ODL2 startup > > cookie=0x1080000, duration=3104.003s, table=19, n_packets=40, > n_bytes=1680, priority=100,arp,arp_op=2 actions=CONTROLLER:65535,resub > mit(,17) > > https://jenkins.opendaylight.org/releng/view/netvirt-csit/jo > b/netvirt-csit-3node-openstack-newton-upstream-learn-boron/ > 55/robot/report/log.html#s1-s3-s2-t27-k5-k1-k1-k1-k16-k3 > > > > After ODL1+ODL2 startup – resubmit is missing > > cookie=0x1080000, duration=0.227s, table=19, n_packets=40, n_bytes=1680, > priority=100,arp,arp_op=2 actions=CONTROLLER:65535 > > https://jenkins.opendaylight.org/releng/view/netvirt-csit/jo > b/netvirt-csit-3node-openstack-newton-upstream-learn-boron/ > 55/robot/report/log.html#s1-s3-s2-t28-k3-k1-k1-k1-k16 > > > > 2. ARP responder flow for router interface in ARP_RESPONDER table > > Before ODL1+ODL2 startup > > cookie=0x122c3d7e, duration=8603.194s, table=81, n_packets=1, n_bytes=42, > priority=100,arp,metadata=0x70337900000222e6/0xffffff00fffff > ffe,arp_tpa=10.10.10.4,arp_op=1 actions=move:NXM_OF_ETH_SRC[]- > >NXM_OF_ETH_DST[],set_field:fa:16:3e:0d:c3:35->eth_src,load: > 0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[] > ,move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0dc3 > 35->NXM_NX_ARP_SHA[],load:0xa0a0a04->NXM_OF_ARP_SPA[], > load:0->NXM_OF_IN_PORT[],load:0x337900->NXM_NX_REG6[],write_ > metadata:0/0x1,goto_table:220 > > cookie=0x62220003, duration=3182.514s, table=81, n_packets=2, n_bytes=84, > priority=100,arp,metadata=0x70000100000251c2/0xffffff00fffffffe,arp_tpa=90.0.0.1,arp_op=1 > actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa > :16:3e:c7:33:48->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_ > NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_ > OF_ARP_TPA[],load:0xfa163ec73348->NXM_NX_ARP_SHA[],load: > 0x5a000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load: > 0x100->NXM_NX_REG6[],resubmit(,220) > > cookie=0x6c22000a, duration=3177.981s, table=81, n_packets=4, n_bytes=168, > priority=100,arp,metadata=0x70000800000251c2/0xffffff00fffffffe,arp_tpa=100.0.0.1,arp_op=1 > actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa > :16:3e:7f:b9:6d->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_ > NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_ > OF_ARP_TPA[],load:0xfa163e7fb96d->NXM_NX_ARP_SHA[],load: > 0x64000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load: > 0x800->NXM_NX_REG6[],resubmit(,220) > > cookie=0x6c220005, duration=3177.909s, table=81, n_packets=4, n_bytes=168, > priority=100,arp,metadata=0x70000300000251c2/0xffffff00fffffffe,arp_tpa=100.0.0.1,arp_op=1 > actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa > :16:3e:7f:b9:6d->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_ > NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_ > OF_ARP_TPA[],load:0xfa163e7fb96d->NXM_NX_ARP_SHA[],load: > 0x64000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load: > 0x300->NXM_NX_REG6[],resubmit(,220) > > https://jenkins.opendaylight.org/releng/view/netvirt-csit/jo > b/netvirt-csit-3node-openstack-mitaka-upstream-learn-carbon/ > 60/robot/report/log.html#s1-s3-s2-t27-k5-k1-k2-k1-k16-k3 > > > > After ODL1+ODL2 startup – flows for arp_tpa 90.0.0.1/100.0.0.1 all nicira > actions are missing > > cookie=0x122c3d7e, duration=9636.373s, table=81, n_packets=1, n_bytes=42, > priority=100,arp,metadata=0x70337900000222e6/0xffffff00fffff > ffe,arp_tpa=10.10.10.4,arp_op=1 actions=move:NXM_OF_ETH_SRC[]- > >NXM_OF_ETH_DST[],set_field:fa:16:3e:0d:c3:35->eth_src,load: > 0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[] > ,move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xfa163e0dc3 > 35->NXM_NX_ARP_SHA[],load:0xa0a0a04->NXM_OF_ARP_SPA[], > load:0->NXM_OF_IN_PORT[],load:0x337900->NXM_NX_REG6[],write_ > metadata:0/0x1,goto_table:220 > > cookie=0x6c22000a, duration=4211.160s, table=81, n_packets=4, n_bytes=168, > priority=100,arp,metadata=0x70000800000251c2/0xffffff00fffffffe,arp_tpa=100.0.0.1,arp_op=1 > actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa > :16:3e:7f:b9:6d->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_ > NX_ARP_SHA[]->NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]->NXM_ > OF_ARP_TPA[],load:0xfa163e7fb96d->NXM_NX_ARP_SHA[],load: > 0x64000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load: > 0x800->NXM_NX_REG6[],resubmit(,220) > > cookie=0x62220003, duration=7.947s, table=81, n_packets=2, n_bytes=84, > priority=100,arp,metadata=0x70000100000251c2/0xffffff00fffffffe,arp_tpa=90.0.0.1,arp_op=1 > actions=set_field:fa:16:3e:c7:33:48->eth_src > > cookie=0x6c220005, duration=7.947s, table=81, n_packets=4, n_bytes=168, > priority=100,arp,metadata=0x70000300000251c2/0xffffff00fffffffe,arp_tpa=100.0.0.1,arp_op=1 > actions=set_field:fa:16:3e:7f:b9:6d->eth_src > > https://jenkins.opendaylight.org/releng/view/netvirt-csit/jo > b/netvirt-csit-3node-openstack-mitaka-upstream-learn-carbon/ > 60/robot/report/log.html#s1-s3-s2-t28-k3-k1-k2-k1-k16-k3 > > > > In the karaf log of ODL3 (that hasn’t been restarted) there are lots of > these errors at the time flows get corrupted > > > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegLoadR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegMoveR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegMoveR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegLoadR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegLoadR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegLoadR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionRegLoadR > pcUpdateFlowUpdatedApplyActionsCase for version 4 > > 2017-01-01 01:55:27,206 | WARN | ult-dispatcher-7 | > ConvertorProcessor | 307 - org.opendaylight.openflowplugin > - 0.4.0.SNAPSHOT | Failed to process interface > org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin > .extension.nicira.action.rev140714.update.flow.input. > updated.flow.instructions.instruction.instruction.apply. > actions._case.apply.actions.action.action.NxActionResubmit > RpcUpdateFlowUpdatedApplyActionsCase for version 4 > > > > https://logs.opendaylight.org/releng/jenkins092/netvirt-csit > -3node-openstack-newton-upstream-learn-boron/55/archives/odl3_karaf.log.gz > > https://logs.opendaylight.org/releng/jenkins092/netvirt-csit > -3node-openstack-mitaka-upstream-learn-carbon/60/archives/ > odl3_karaf.log.gz > > > > Has anyone encountered this before or have any idea how to fix it? > > > > Thanks > > Tali > > > _______________________________________________ > openflowplugin-dev mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev > > >
_______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
