Hi Jamo, Thanks for the analysis, as I commented in private to some openflow committers the openflowplugin main feature (flow services) is "not experimental" in single node:
https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-flow-services-only-boron/ <https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-flow-services-only-boron/> However the same feature is "experimental" when run in cluster environment: https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/openflowplugin-csit-3node-clustering-only-boron/ <https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/openflowplugin-csit-3node-clustering-only-boron/> My guess is that most of the cluster instabilities are due to blocker bug: https://bugs.opendaylight.org/show_bug.cgi?id=6554 <https://bugs.opendaylight.org/show_bug.cgi?id=6554> So if we solve the above in the coming days there are chances for the openflow cluster to be also "not experimental". For your comments see in-line: > On Aug 31, 2016, at 10:51 PM, Jamo Luhrsen <[email protected]> wrote: > > For the OpenflowPlugin release review Thursday morning, I have the following > analysis of their > upstream CSIT for boron, using the most recent boron job result. > > please note that I do not know if all delivered features have system test or > not, so > only reporting on what exists... which is a LOT! > > It's hard to know what's really happening here. I think the main > functionality suite "flow-services" > is passing 100% and probably gives some confidence. But with the other > suites having what looks > like basic issues, I am a bit worried. So, just reporting for now. I have > some extra details > below the job listing. > > NOT-OK 3node-periodic-bulkomatic-clustering-daily-only-boron > (unexpected failures) > NOT-OK 3node-periodic-bulkomatic-clustering-daily-helium-redesign-only-boron > (unexpected failures) > NOT-OK 3node-clustering-only-boron > (unexpected failures) > NOT-OK 3node-clustering-helium-redesign-only-boron > (unexpected failures) > NOT-OK 1node-scalability-helium-redesign-only-boron > (unexpected failures) > NOT-OK > 1node-periodic-scale-stats-collection-daily-helium-redesign-only-boron > (unexpected failures) > NOT-OK 1node-periodic-scale-stats-collection-daily-frs-only-boron > (unexpected failures) > NOT-OK 1node-periodic-scalability-daily-helium-redesign-only-boron > (scale test found zero) > NOT-OK 1node-periodic-longevity-only-boron > (unexpected failures) > NOT-OK 1node-periodic-longevity-helium-redesign-only-boron > (unexpected failures) > NOT-OK 1node-periodic-link-scalability-daily-helium-redesign-only-boron > (scale test found zero) > NOT-OK 1node-flow-services-helium-redesign-only-boron > (unexpected failures) > NOT-OK 1node-flow-services-frs-only-boron > (unexpected failures) > > > OK 1node-scalability-only-boron > OK 1node-periodic-sw-scalability-daily-only-boron > (scaled to 500 switches) > OK 1node-periodic-sw-scalability-daily-helium-redesign-only-boron > (scaled to 500 switches) > OK 1node-periodic-scale-stats-collection-daily-only-boron > OK 1node-periodic-rpc-time-measure-daily-only-boron > OK 1node-periodic-rpc-time-measure-daily-helium-redesign-only-boron > OK 1node-periodic-link-scalability-daily-only-boron > (??scaling to ~2500 links) > OK 1node-periodic-cbench-daily-only-boron > (critical bug found here) > OK 1node-periodic-cbench-daily-helium-redesign-only-boron > (perf test found zero) > OK 1node-periodic-bulkomatic-perf-daily-only-boron > OK 1node-periodic-bulk-matic-ds-daily-only-boron > OK 1node-periodic-bulk-matic-ds-daily-helium-redesign-only-boron > OK 1node-flow-services-only-boron > OK 1node-flow-services-all-boron > OK 1node-config-performance-only-boron > OK 1node-config-performance-helium-redesign-only-boron > OK 1node-cbench-performance-only-boron > (critical bug found here) > OK 1node-cbench-performance-helium-redesign-only-boron > (perf test found zero) > > Some failures I saw actually pointed clearly to a bug, but the bug was in > resolved state so > that means it's a new type of failure, or we have a regression. Can you tell where do you see this? > > Some failures might be in perf/scale related tests and we may decide that > those failures > are ok because they are relative levels we are testing against. However, > some of the failures > I saw in performance jobs looked to be fundamental (e.g. zero flows found) Is this the Cbench throughput test? if so we have already a critical bug. > > There were failures in longevity tests that were also worrisome because of > how short the job > ran before failing. Seems something basic is breaking there. The default > plugin longevity > job has a thread count graph that was up and to the right over time and made > me think about > a thread leak. The helium plugin never even saw the first group of connected > switches > and failed straight away. The first could be a critical bug but we never got that far fixing more fundamental issues to pay attention to this. The second is because helium plugin LLDP speaker does not work in boron and probably will not be fixed because this feature is not shipped as default plugin. > > The cbench test fails and points to a bug that is non-blocking, but critical. > Per our > expectations this is still ok. I urge openflowplugin to double check if it > should be > upgraded to blocker or not, but I am sure they have scrubbed it more than > once already. In my opinion this should be a blocker because 1) it is a regression from Beryllium and 2) anybody testing ODL perf will hit this issue and whatever perf report will come after will harm ODL. > > The -frs-only-boron suite looks like its having major troubles interacting > with a tools > vm. I didn't dig too deep, but that's my first guess. > > > Thanks, > JamO > >
_______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
