Hi Jamo, FYI, -frs-only-boron suites were added recently (26/8) to test new feature added to OFP during boron dev phase. Not sure about your first guess, but there is also one bug 6542 [1] (already with proposed patch), but not merged yet cause it was found already in only-blocker phase.
[1] https://bugs.opendaylight.org/show_bug.cgi?id=6542 ________________________________________ From: Jamo Luhrsen <[email protected]> Sent: Thursday, September 1, 2016 7:51 AM To: [email protected] Cc: An Ho; Colin Dixon; [email protected] Subject: [integration-dev] OpenflowPlugin CSIT analysis for release Boron review (9/1) For the OpenflowPlugin release review Thursday morning, I have the following analysis of their upstream CSIT for boron, using the most recent boron job result. please note that I do not know if all delivered features have system test or not, so only reporting on what exists... which is a LOT! It's hard to know what's really happening here. I think the main functionality suite "flow-services" is passing 100% and probably gives some confidence. But with the other suites having what looks like basic issues, I am a bit worried. So, just reporting for now. I have some extra details below the job listing. NOT-OK 3node-periodic-bulkomatic-clustering-daily-only-boron (unexpected failures) NOT-OK 3node-periodic-bulkomatic-clustering-daily-helium-redesign-only-boron (unexpected failures) NOT-OK 3node-clustering-only-boron (unexpected failures) NOT-OK 3node-clustering-helium-redesign-only-boron (unexpected failures) NOT-OK 1node-scalability-helium-redesign-only-boron (unexpected failures) NOT-OK 1node-periodic-scale-stats-collection-daily-helium-redesign-only-boron (unexpected failures) NOT-OK 1node-periodic-scale-stats-collection-daily-frs-only-boron (unexpected failures) NOT-OK 1node-periodic-scalability-daily-helium-redesign-only-boron (scale test found zero) NOT-OK 1node-periodic-longevity-only-boron (unexpected failures) NOT-OK 1node-periodic-longevity-helium-redesign-only-boron (unexpected failures) NOT-OK 1node-periodic-link-scalability-daily-helium-redesign-only-boron (scale test found zero) NOT-OK 1node-flow-services-helium-redesign-only-boron (unexpected failures) NOT-OK 1node-flow-services-frs-only-boron (unexpected failures) OK 1node-scalability-only-boron OK 1node-periodic-sw-scalability-daily-only-boron (scaled to 500 switches) OK 1node-periodic-sw-scalability-daily-helium-redesign-only-boron (scaled to 500 switches) OK 1node-periodic-scale-stats-collection-daily-only-boron OK 1node-periodic-rpc-time-measure-daily-only-boron OK 1node-periodic-rpc-time-measure-daily-helium-redesign-only-boron OK 1node-periodic-link-scalability-daily-only-boron (??scaling to ~2500 links) OK 1node-periodic-cbench-daily-only-boron (critical bug found here) OK 1node-periodic-cbench-daily-helium-redesign-only-boron (perf test found zero) OK 1node-periodic-bulkomatic-perf-daily-only-boron OK 1node-periodic-bulk-matic-ds-daily-only-boron OK 1node-periodic-bulk-matic-ds-daily-helium-redesign-only-boron OK 1node-flow-services-only-boron OK 1node-flow-services-all-boron OK 1node-config-performance-only-boron OK 1node-config-performance-helium-redesign-only-boron OK 1node-cbench-performance-only-boron (critical bug found here) OK 1node-cbench-performance-helium-redesign-only-boron (perf test found zero) Some failures I saw actually pointed clearly to a bug, but the bug was in resolved state so that means it's a new type of failure, or we have a regression. Some failures might be in perf/scale related tests and we may decide that those failures are ok because they are relative levels we are testing against. However, some of the failures I saw in performance jobs looked to be fundamental (e.g. zero flows found) There were failures in longevity tests that were also worrisome because of how short the job ran before failing. Seems something basic is breaking there. The default plugin longevity job has a thread count graph that was up and to the right over time and made me think about a thread leak. The helium plugin never even saw the first group of connected switches and failed straight away. The cbench test fails and points to a bug that is non-blocking, but critical. Per our expectations this is still ok. I urge openflowplugin to double check if it should be upgraded to blocker or not, but I am sure they have scrubbed it more than once already. The -frs-only-boron suite looks like its having major troubles interacting with a tools vm. I didn't dig too deep, but that's my first guess. Thanks, JamO _______________________________________________ integration-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/integration-dev AndrejLeitner Software Developer Sídlo / Mlynské Nivy 56 / 821 05 Bratislava / Slovakia R&D centrum / Janka Kráľa 9 / 974 01 Banská Bystrica / Slovakia / [email protected] reception: +421 2 206 65 114 / www.pantheon.sk [logo] _______________________________________________ openflowplugin-dev mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
