John -- I agree with you entirely here. My concern is more that I think the CI tests need to run more frequently than weekly.
Michael On Thu, Jan 16, 2014 at 9:30 AM, John Griffith <john.griff...@solidfire.com> wrote: > On Wed, Jan 15, 2014 at 6:03 PM, Michael Still <mi...@stillhq.com> wrote: >> On Thu, Jan 16, 2014 at 6:28 AM, John Griffith >> <john.griff...@solidfire.com> wrote: >>> Hey Everyone, >>> >>> A while back I started talking about this idea of requiring Cinder >>> driver contributors to run a super simple cert script (some info here: >>> [1]). Since then I've been playing with introduction of a third party >>> gate check here in my own lab. My proposal was to have a non-voting >>> check that basically duplicates the base devstack gate test in my lab, >>> but uses different back-end devices that I have available configured >>> in Cinder to run periodic tests against. Long term I'd like to be >>> able to purpose this gear to also do something "more useful" for the >>> over all OpenStack gating effort but to start it's strictly an >>> automated verification of my Cinder driver/backend. >>> >>> What I'm questioning is how to report this information and the >>> results. Currently patches and reviews are our mechanism for >>> triggering tests and providing feedback. Myself and many other >>> vendors that might like to participate in something like this >>> obviously don't have the infrastructure to try and run something like >>> this on every single commit. Also since it would be non-voting it's >>> difficult to capture and track the results. >>> >>> One idea that I had was to set something like what I've described >>> above to run locally on a periodic basis (weekly, nightly etc) and >>> publish results to something like a "third party verification >>> dashboard". So the idea would be that results from various third >>> party tests would all adhere to a certain set of criteria WRT what >>> they do and what they report and those results would be logged and >>> tracked publicly for anybody in the OpenStack community to access and >>> view? >> >> My concern here is how to identify what patch broke the third party >> thing. If you run this once a week, then there are possible hundreds >> of patches which might be responsible. How do you identify which one >> is the winner? > > To be honest I'd like to see more than once a week, however the main > point of this is to have public testing of third party drivers. > Currently we say "it's in trunk and passed review and unit tests" so > you're good to go. Frankly that's not sufficient, there needs to be > some sort of testing publicly that shows that a product/config > actually works in the minimum sense at least. This won't address > things like a bad patch breaking things, but again in Cinder's case > this is a bit different, it is designed more to show compatibility and > integration completeness. If a patch goes in and breaks a vendors > driver but not the reference implementation, that means the vendor has > work to do bring their driver up to date. > > Cinder is not a dumping ground, the drivers in the code base should no > be static but require continued maintenance and development as the > project grows. > > Non-Voting tests on every patch seems unrealistic, however there's no > reason that if vendors have the resources they couldn't do that if > they so choose. > >> >>> Does this seem like something that others would be interested in >>> participating in? I think it's extremely valuable for projects like >>> Cinder that have dozens of backend devices, and regardless of other >>> interest or participation in the community I intend to implement >>> something like this on my own regardless. It would just be >>> interesting to see if we could have an organized and official effort >>> to gather this sort of information and run these types of tests. >>> >>> Open to suggestions and thoughts as well as any of you that may >>> already be doing this sort of thing. By the way, I've been looking at >>> things like SmokeStack and other third party gating checks to get some >>> ideas as well. >> >> Michael >> >> -- >> Rackspace Australia >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Rackspace Australia _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev