Re: [openstack-dev] [Manila] Ask for help on supportting the 3-rd party CI for HDFS driver
On 06/11/2015 10:34 AM, Jeremy Stanley wrote: On 2015-06-11 07:51:55 +0200 (+0200), Philipp Marek wrote: [...] I still stand by my opinion (as voiced in Vancouver) that for such one-off things (that contributors are not likely to repeat over and over again) it might make sense to have -infra simply *do* them[3]. [...] To reiterate my response from the summit, it's a neat idea but not one the Infra team has the bandwidth to entertain at the moment. As you've noticed we're understaffed and while we're continually trying to grow the team it takes many months to a year or more of full-time exposure to our current systems to bring new people up to speed to help us run it. Also we don't actually have a holistic view of the underlying tests being run by the jobs... for that you need to elicit assistance from the QA team who maintain DevStack/Tempest and did the plugin design for things like out-of-tree driver testing, and also the project teams for the software at which these drivers and backends are targeted. So while I and others are happy to have our CI run jobs to test OpenStack drivers for other free software backends, don't expect the actual work and learning curve to necessarily be any less than building your own CI system from scratch (just different). It doesn't make sense to require people to learn about things they will never use again - and the amount of time spent answering the questions, diagnosing problems and so on is quite a bit higher than doing it simply right the first time. This is, I think, also a common misconception. The people who write these jobs to run in our CI need to stick around or induct successors to help maintain them and avoid bitrot as our systems constantly change and evolve. I know the same goes for the drivers themselves... if people don't keep them current with the OpenStack software into which they're integrating, support will quickly be dropped due to quality control concerns. I strongly agree here. I think that the cinder community has shown that the one of the main values of universal vendor CI is that it keeps driver maintainers engaged with the community and aware of ongoing development. OpenStack projects are not a static target which you can write a driver for once and be done. We are adding new features and making enhancements every release, and some of those changes require drivers to evolve too. At the very least CI systems allow us to validate that the introduction of a new feature didn't break any existing drivers. For a pure-software based storage backend like HDFS, we can leverage the compute resources of openstack-infra, but the development resources still need to come from the Manila team -- the same group people responsible for maintaining the driver and fixing bugs should have some understanding of the automated test system because it will be finding bugs and we'll have to reproduce failure and debug them. If nobody is willing to do this on an ongoing basis for a backend like HDFS, then eventually we won't be able to support it anymore. The CI requirement just makes this fact more explicit and forces us to either commit the resources or remove the driver rather than waiting until the driver is horribly broken in a few years. And if it's *that* often needed, why not write a small script that, given a name, does the needed changes, so that only a commit review is needed? [...] Definitely something that people who have experience writing these could collaborate on contributing. As I mentioned, the Infra team doesn't have the complete picture, but the people who have sweated and bled to get their drivers tested and integrated do, at least to a greater extent than we do. This is all to say I understand the frustration, but I don't have a simple solution for it unfortunately. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Manila] Ask for help on supportting the 3-rd party CI for HDFS driver
It doesn't make sense to require people to learn about things they will never use again - and the amount of time spent answering the questions, diagnosing problems and so on is quite a bit higher than doing it simply right the first time. This is, I think, also a common misconception. The people who write these jobs to run in our CI need to stick around or induct successors to help maintain them and avoid bitrot as our systems constantly change and evolve. I know the same goes for the drivers themselves... if people don't keep them current with the OpenStack software into which they're integrating, support will quickly be dropped due to quality control concerns. Maintaining and supporting the drivers isn't the problem - that has to happen more or less regularly, if only to add features and/or to stay compatible with the rest of the Openstack code. It's also much nearer to the contributor's normal work, so not that likely to be forgotten - whereas the job definitions are looked at (perhaps) never again, and so the knowledge about that will be lost (from the contributor's brain, that is). To repeat the important point: Thanks to all people that are helping on IRC, this is one of the most important assets Openstack has today! -- : Ing. Philipp Marek : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com : DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Manila] Ask for help on supportting the 3-rd party CI for HDFS driver
On 2015-06-11 07:51:55 +0200 (+0200), Philipp Marek wrote: [...] I still stand by my opinion (as voiced in Vancouver) that for such one-off things (that contributors are not likely to repeat over and over again) it might make sense to have -infra simply *do* them[3]. [...] To reiterate my response from the summit, it's a neat idea but not one the Infra team has the bandwidth to entertain at the moment. As you've noticed we're understaffed and while we're continually trying to grow the team it takes many months to a year or more of full-time exposure to our current systems to bring new people up to speed to help us run it. Also we don't actually have a holistic view of the underlying tests being run by the jobs... for that you need to elicit assistance from the QA team who maintain DevStack/Tempest and did the plugin design for things like out-of-tree driver testing, and also the project teams for the software at which these drivers and backends are targeted. So while I and others are happy to have our CI run jobs to test OpenStack drivers for other free software backends, don't expect the actual work and learning curve to necessarily be any less than building your own CI system from scratch (just different). It doesn't make sense to require people to learn about things they will never use again - and the amount of time spent answering the questions, diagnosing problems and so on is quite a bit higher than doing it simply right the first time. This is, I think, also a common misconception. The people who write these jobs to run in our CI need to stick around or induct successors to help maintain them and avoid bitrot as our systems constantly change and evolve. I know the same goes for the drivers themselves... if people don't keep them current with the OpenStack software into which they're integrating, support will quickly be dropped due to quality control concerns. And if it's *that* often needed, why not write a small script that, given a name, does the needed changes, so that only a commit review is needed? [...] Definitely something that people who have experience writing these could collaborate on contributing. As I mentioned, the Infra team doesn't have the complete picture, but the people who have sweated and bled to get their drivers tested and integrated do, at least to a greater extent than we do. This is all to say I understand the frustration, but I don't have a simple solution for it unfortunately. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Manila] Ask for help on supportting the 3-rd party CI for HDFS driver
Hi Chen Li, You are correct in that setting up a CI system is not a trivial task. IMO it would make sense to have this eventually tested with infras infrastructure but as Jeremy mentioned, they don't have the bandwidth to do the setup. Below are some links to get started if you all are interested in setting up a CI system. If this is not possible I recommend bringing it up in the Manila weekly meeting so Manila folks can figure out how to get the HDFS system tested and perhaps this is work that Manila core can accomplish. OpenStack Third Party CI requirements: http://docs.openstack.org/infra/system-config/third_party.html Third Party CI meetings: https://wiki.openstack.org/wiki/Meetings/ThirdParty (Mondays at 1500 UTC and Tuesdays at 0800 UTC are the meetings to get help with CI setup) Tools for setting up CI: https://github.com/openstack-infra/puppet-openstackci/ (this is just getting started) https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers (Cinder specific but helpful) -Alex On Wed, Jun 10, 2015 at 3:48 AM, Li, Chen chen...@intel.com wrote: Hello Manila, There is a HDFS driver in Manila summited by our team in Kilo, so I guess we do own this driver in Manila for current situation. But, CI systems are really new to us, and we heard from other teams that they took almost one year to implement a CI for going through all the company processes and building all related staff for it. And , the biggest issue we have is there is not enough resource to actually implement and maintain the CI as that is not allowed by the team’s strategy. We strongly believe the HDFS driver will be really useful in Manila/Openstack, and might be used by other Openstack projects such as Sahara/Cognitive which are big data related projects in the near future, thus we don’t want to see the driver to be remove. Do we have a volunteer to take over the CI for HDFS driver as it is an open source distributed storage system which actually has no vendor dependency ? Looking forward to your reply. Thanks. -chen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Manila] Ask for help on supportting the 3-rd party CI for HDFS driver
Hello Manila, There is a HDFS driver in Manila summited by our team in Kilo, so I guess we do own this driver in Manila for current situation. But, CI systems are really new to us, and we heard from other teams that they took almost one year to implement a CI for going through all the company processes and building all related staff for it. And , the biggest issue we have is there is not enough resource to actually implement and maintain the CI as that is not allowed by the team's strategy. We strongly believe the HDFS driver will be really useful in Manila/Openstack, and might be used by other Openstack projects such as Sahara/Cognitive which are big data related projects in the near future, thus we don't want to see the driver to be remove. Do we have a volunteer to take over the CI for HDFS driver as it is an open source distributed storage system which actually has no vendor dependency ? Looking forward to your reply. Thanks. -chen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Manila] Ask for help on supportting the 3-rd party CI for HDFS driver
Hi Chen, You can ask if OpenStack Infrastructure team can set up CI for this driver. They have set up CI's for a few Cinder drivers based on open source storage systems. Thanks, Xing From: Li, Chen [mailto:chen...@intel.com] Sent: Wednesday, June 10, 2015 3:48 AM To: OpenStack Development Mailing List (not for usage questions) (openstack-dev@lists.openstack.org) Subject: [openstack-dev] [Manila] Ask for help on supportting the 3-rd party CI for HDFS driver Hello Manila, There is a HDFS driver in Manila summited by our team in Kilo, so I guess we do own this driver in Manila for current situation. But, CI systems are really new to us, and we heard from other teams that they took almost one year to implement a CI for going through all the company processes and building all related staff for it. And , the biggest issue we have is there is not enough resource to actually implement and maintain the CI as that is not allowed by the team's strategy. We strongly believe the HDFS driver will be really useful in Manila/Openstack, and might be used by other Openstack projects such as Sahara/Cognitive which are big data related projects in the near future, thus we don't want to see the driver to be remove. Do we have a volunteer to take over the CI for HDFS driver as it is an open source distributed storage system which actually has no vendor dependency ? Looking forward to your reply. Thanks. -chen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Manila] Ask for help on supportting the 3-rd party CI for HDFS driver
On 2015-06-10 14:39:46 + (+), yang, xing wrote: You can ask if OpenStack Infrastructure team can set up CI for this driver. They have set up CI's for a few Cinder drivers based on open source storage systems. To be fair, the Infra team hasn't really written the jobs to test those, but we have pointed the developers responsible for the drivers to our documentation and reviewed the changes they submit to add the jobs to our CI. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Manila] Ask for help on supportting the 3-rd party CI for HDFS driver
Thanks Jeremy. I assume Chen could follow this example to add a job for the HDFS driver? https://review.openstack.org/#/c/188744/ Thanks, Xing -Original Message- From: Jeremy Stanley [mailto:fu...@yuggoth.org] Sent: Wednesday, June 10, 2015 11:40 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Manila] Ask for help on supportting the 3-rd party CI for HDFS driver On 2015-06-10 14:39:46 + (+), yang, xing wrote: You can ask if OpenStack Infrastructure team can set up CI for this driver. They have set up CI's for a few Cinder drivers based on open source storage systems. To be fair, the Infra team hasn't really written the jobs to test those, but we have pointed the developers responsible for the drivers to our documentation and reviewed the changes they submit to add the jobs to our CI. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Manila] Ask for help on supportting the 3-rd party CI for HDFS driver
On 2015-06-10 18:20:24 + (+), yang, xing wrote: Thanks Jeremy. I assume Chen could follow this example to add a job for the HDFS driver? https://review.openstack.org/#/c/188744/ That's a fine short-form answer. The longer answer is to solicit input from some of the people who have done similar work, so that mistakes can be learned from rather than repeated. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Manila] Ask for help on supportting the 3-rd party CI for HDFS driver
Hi all, Thanks Jeremy. I assume Chen could follow this example to add a job for the HDFS driver? https://review.openstack.org/#/c/188744/ That's a fine short-form answer. The longer answer is to solicit input from some of the people who have done similar work, so that mistakes can be learned from rather than repeated. As requested, here's some input from me. Oh well, where to start? Perhaps at the beginning ... Openstack has quite a lot of documentation - in Wiki pages, code examples, and so on. Navigating all of that (well, even finding the relevant pieces) is not that easy - and some parts are wrong in details or contradict each other. Unavoidable for a project of this size, but still annoying for newcomers. What I'd very much have liked to see was some big-picture overview how the processes and configurations work[1]. Some short description what the files in the project-config mean, what they are used for, etc., to get a basic understanding how that works _without_ needing to learn all the auxillary tools (of which there are quite a few - Zuul, Jenkins, Puppet, ...). What *I* did get wrong (and this an important point to learn from!) is that even things like the devstack plugin name matters. This isn't written anywhere, but if you get it wrong then the various jobs won't match the generated check gate scriptlet names anymore, and then there are some more hoops to jump through...[2] I still stand by my opinion (as voiced in Vancouver) that for such one-off things (that contributors are not likely to repeat over and over again) it might make sense to have -infra simply *do* them[3]. But let's get back to the topic - what I have learned, resp. what I did wrong, so that others can learn. The quoted change number above is not enough; I've pushed a few updates already, so to get a more complete example it might be better to take a checkout and to filter by my changes: # git log --committer philipp.ma...@linbit.com Looking at the *combined* diff of that might be a first step. Or perhaps not, because of the wrong name... Another idea that I can pass on is that as soon as you've had the first CI run, navigate the logfile directory to fetch the local.conf and tempest.conf file (which will be compressed), and to look at them. Some small errors might be diagnosed from there[4], without needing to spend time looking what goes wrong and why. Anything else? Hmmm... yeah, the last point I can mention is that people on IRC (-infra, for that topic) are very helpful. If you're unlucky, they're stressed by other problems and won't have that much time; but generally, you'll receive answers quite fast. (Unless you're in the wrong timezone. AJaeger is very helpful in the CET zones - but he's alone [I guess], and so it's a game of luck again. Having some more -infra cores distributed around the globe would be nice.) Well - you have to know to ask the right questions, right. While sometimes you'll get extensive answers, in busier times you might get the *exact* answer to your question - whether it's helpful in the long term, or not. Overall, it's been both more awful than expected, and at the same time more people help than with other Open-Source projects; I guess I'll have to stop here, to avoid starting a rant about another topic. I hope that helps - at least a little bit. Regards, Phil == Ad 1: In the last 8 weeks (since starting the CI in -infra) I got some ideas; I'm not sure how much is correct, but if there's interest, I can try to put them down into a wiki (please point me to some suitable location). Ad 2: Another thing that is not written down (but could be guessed) is that some lists have to be kept in alphabetical order... Ad 3: Eg. for free if it's an Open Source project, and for a small fee like $200 or so for proprietary ones - that's still some major savings for the company, compared to spending tens of man-hours trying to get that right. It doesn't make sense to require people to learn about things they will never use again - and the amount of time spent answering the questions, diagnosing problems and so on is quite a bit higher than doing it simply right the first time. And if it's *that* often needed, why not write a small script that, given a name, does the needed changes, so that only a commit review is needed? Ad 4: Eg., the cinder tests multi-backend tests failed for my driver. After looking at tempest.conf I figured out (thanks, jgriffith, for telling me that too) that they were *wrongly* activated - because I've had - CINDER_ENABLED_BACKENDS=,drbd:drbdmanage + CINDER_ENABLED_BACKENDS=drbd:drbdmanage A small comma in the setup definition - and several tests fail, instead of them getting skipped... -- : Ing. Philipp Marek : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com : DRBD® and LINBIT® are registered trademarks
