Hi Bharat, I’d say remove the tests that fail, to get a report with only tests that pass.
Regards, Remi From: Bharat Kumar <bharat.ku...@accelerite.com<mailto:bharat.ku...@accelerite.com>> Date: Tuesday 15 March 2016 at 10:40 To: Remi Bergsma <rberg...@schubergphilis.com<mailto:rberg...@schubergphilis.com>> Cc: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>, Bharat Kumar <bharat.ku...@accelerite.com<mailto:bharat.ku...@accelerite.com>> Subject: Re: [PROPOSAL] Minimum Viable CI Integration Hi Remi, Thanks for the inputs. I agree with you, May be i need to modify the CI report and not report if any of the known issues fail. We can still have the known_issues header to track what needs to be fixed. Thanks, Bharat. On 15-Mar-2016, at 1:26 PM, Remi Bergsma <rberg...@schubergphilis.com<mailto:rberg...@schubergphilis.com>> wrote: Hi Bharat, I’d suggest removing all tests that fail for some reason, not being an error in the PR. This will result in a smaller set of tests that “always pass”. Once a test doesn’t pass anymore, we know the PR has an issue. Then your CI system provides value. Right now, there are so many failures that soon people don’t even care to look at it. Once we have a stable set that works, we can start fixing tests and 1-by-1 add them to the mix. Just my $0.02 Regards, Remi On 15/03/16 07:06, "Bharat Kumar" <bharat.ku...@accelerite.com<mailto:bharat.ku...@accelerite.com>> wrote: Hi Paul, I totally agree with you. For now we should ignore the tests which fail all the time while we try to address them. Now that we have a CI in place we need go and fix issues with the tests. I have tried to identify a few of them and have mentioned them in the known issues part of the CI test result. Some of the test cases are not generic enough to run on different CIs. For example the test_vpc_vpn.py hardcodes the test data in the test itself, which is not a good thing. Generally this kind of data is added the the test_data.py file and is reused . This gives us a chance to modify it to suit the needs of the a particular CI. So for fixing these issues we need test data to identify the issue. Each of the CI run gives these details, the test run info, test results and the management server logs( via the drop box link in the report). With this info any one in the community can access this data and start fixing them. This way we can progressively track and improve the health of the release. So i think we should continue posting these results, so that people can analyse the result and come back with inputs related to incorrect test cases. —Thanks Bharat. On 15-Mar-2016, at 1:24 AM, Paul Angus <paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com><mailto:paul.an...@shapeblue.com>> wrote: Cool. Thanks Will. @Bharat - the work that you're trying to do is great, but until the Marvin tests have been gone through and 'fixed' the information being put into PRs is misleading. No one knows why these failures are occurring, and it's very likely giving both false positives and false negatives. You should probably hold off altogether or isolate the good tests and only run them. I plan to set up a wiki page (or two) with all of the tests listed, their purpose and other info such as if there are known to work (or not) when they were last checked. You're welcome to make a start on it if you wish. For example (false negative): ------------------------------ nosetests --with-marvin --marvin-config=/tmp/te2.cfg --hypervisor=KVM -a tags=advanced,required_hardware=true integration/component/test_acl_isolatednetwork.py 2016-01-30 07:36:30,488 - DEBUG - Parsing Test data successful 2016-01-30 07:36:30,488 - DEBUG - Payload: {'account': 'admin', 'command': 'listUsers', 'response': 'json'} 2016-01-30 07:36:30,488 - DEBUG - ========Sending GET Cmd : listUsers======= 2016-01-30 07:36:30,521 - DEBUG - Response : [{username : u'admin', account : u'admin', domainid : u'4895ddbe-bfac-11e5-be38-06a910012727', firstname : u'admin', created : u'2016-01-20T19:33:19+0000', lastname : u'cloud', apikey : u'oiD-zcjZw7dsPq8yrjarls3sCtHV0E8FQYRpNXlmEx0ssJ1fKFuCySYL4vLHVmlLA5E3PF7_hSnTND_eSRyK_A', domain : u'ROOT', secretkey : u'gLz6lDVFch6ayBAB-8SMZ-RPiOgJTEVJmNgnAMdv36O6eMUKUN6R-Q-3ZtYJUH84jye-5q6Qpxf5WOtSuG8WtA', iscallerchilddomain : False, state : u'enabled', accounttype : 1, id : u'a8b21230-bfac-11e5-be38-06a910012727', isdefault : True, accountid : u'a8b2063c-bfac-11e5-be38-06a910012727'}] 2016-01-30 07:36:30,521 - DEBUG - ==== Test Client Creation Successful ==== ----------------------------------- Nothing actually happens ?! Same with test_acl_listvm.py Example(false positive-failure is in marvin test code): ------------------------------------------------------------- nosetests --with-marvin --marvin-config=/tmp/te2.cfg --hypervisor=KVM -a tags=advanced,required_hardware=true /tmp/integration/component/test_browse_templates.py 2016-01-30 19:52:26,160 - CRITICAL - EXCEPTION: None: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/nose/suite.py", line 209, in run\n self.setUp()\n', ' File "/usr/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp\n self.setupContext(ancestor)\n', ' File "/usr/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext\n try_run(context, names)\n', ' File "/usr/lib/python2.7/site-packages/nose/util.py", line 471, in try_run\n return func()\n', ' File "/tmp/integration/component/test_browse_templates.py", line 84, in setUpClass\n cls.uploadurl=cls.testdata["configurableData"]["browser_upload_template"][cls.uploadtemplateformat]["url"]\n', "KeyError: 'browser_upload_template'\n"] <http://www.shapeblue.com/> Paul Angus VP Technology , ShapeBlue d: +44 203 617 0528 | s: +44 203 603 0540<tel:+44%20203%20617%200528%20|%20s:%20+44%20203%20603%200540> | m: +44 7711 418784<tel:+44%207711%20418784> e: paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com> | t: @cloudyangus<mailto:paul.an...@shapeblue.com%20|%20t:%20@cloudyangus> | w: www.shapeblue.com<http://www.shapeblue.com/><http://www.shapeblue.com/> a: 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark. This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. -----Original Message----- From: Will Stevens [mailto:williamstev...@gmail.com] Sent: Monday, March 14, 2016 4:26 AM To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org><mailto:dev@cloudstack.apache.org> Subject: Re: [PROPOSAL] Minimum Viable CI Integration The latest version (0.2.0) of upr is now available... https://github.com/cloudops/upr All the details are in the README file of the repo, but the quick overview of the changes include: - Added the ability to comment on a pull request (by PR number or commit). - Implemented both S3 and Swift object storage backends for uploading files (tested with AWS S3, Cloudian (S3) and Swift). - Allow for automatic expiration of uploaded files to clean up the object store after a period of time (in both S3 and Swift). - Added the ability to override the default comment template with a local file (look at the current template for details). Let me know if you guys have questions, comments or feedback on that... Hopefully the apache infra guys will get back to me tomorrow so we can better assess how we need to move forward. Cheers, Will On Sun, Mar 13, 2016 at 7:11 PM, Will Stevens <williamstev...@gmail.com<mailto:williamstev...@gmail.com><mailto:williamstev...@gmail.com>> wrote: Unfortunately this would only give us the ability to 'red light'/'green light' pull requests via a distributed CI environments. The reason I limited the request to this permission is because anything they give us will be global to the entire apache org and this permission is restrictive enough that we can't do damage elsewhere. There is a bit of history of them denying all requests of this manner, so I wanted to focus on our minimum viable use case, while limiting their liability, to improve our chances. You can follow the action here: https://issues.apache.org/jira/browse/INFRA-11429 They have officially said no at this point, but I have asked them to reconsider and asked them to provide suggestions for solving our problem. If they say no again, they will be doing so with the full knowledge that the decision leaves us with our hands tied and we have exhausted our official options through them. If we end up in this situation, we will have to get creative and I suspect that will result in some activities moving outside the Apache repo. I have not gone there (yet) since I feel that will need to be voted on in an official thread of its own before we make a decision like that. Feedback and suggestions are very welcome. I am finalizing the 'comment' functionality in my tool 'upr' this weekend, so that could give us some limited options. I will try to push a release later tonight. It is all written and functional now, just need to clean up the documentation and examples. Find out more about ShapeBlue and our range of CloudStack related services: IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> | CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> | CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/> DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails. DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
