Thx Daan. All feedback counts. Positive feedback is also good, it means someone has actually read this and does not completely disagree with me. :)
*Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Wed, Mar 9, 2016 at 11:41 AM, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > Not really Will, Though I like the key per user idea. (positive feedback > doesn't count does it?) > > On Wed, Mar 9, 2016 at 5:21 PM, Will Stevens <wstev...@cloudops.com> > wrote: > > > Anyone have any feedback on this? I would like to get this ticket opened > > this week. > > > > *Will STEVENS* > > Lead Developer > > > > *CloudOps* *| *Cloud Solutions Experts > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > w cloudops.com *|* tw @CloudOps_ > > > > On Tue, Mar 8, 2016 at 12:46 PM, Will Stevens <wstev...@cloudops.com> > > wrote: > > > > > I am going to open a ticket with the infra team to request access > tokens > > > for each of the organizations who are putting up hardware for the CI > > cause. > > > > > > The reason I am planning to request a token for each organization > > > individually is because I want to make a point about our need for CI > and > > > the need for many groups to be doing CI in order for us to get full > > > hardware coverage of the different features we support. The token is > > > needed in order for the different CI implementations to be able to > simply > > > post back a 'red light', 'green light' status of the pull request in > > > github. More details can be added in comments using an implementation > > like > > > what Bharat is working on, but the focus of this request will be to > > enable > > > us to have the ability for the different CI environments to get at > least > > > 'red light', 'green light' functionality for pull requests. > > > > > > I will be requesting a token with the following basic permissions: > > > - public_repo - the same as an anonymous user > > > - repo:status - this allows for the status of a PR to be changed to one > > > of the following 4 states; pending, success, failure, error, with some > > > additional information such as; context, description, and a public url > > for > > > more details of the status. > > > > > > I am going to associate people with each token request and ask that > each > > > person be sent their own token since they are providing infrastructure > to > > > operate a CI environment and the community would like visibility into > > their > > > results. > > > > > > The people I am going to ask for a token for are as follows: > > > > > > Will Stevens <wstev...@cloudops.com> > > > Paul Angus <paul.an...@shapeblue.com> > > > Bharat Kumar <bharat.ku...@accelerite.com> > > > Remi Bergsma <rberg...@schubergphilis.com> > > > > > > If you would also like to be included in this request and get your own > > > token in order to be able to contribute back the status of your CI > runs, > > > please let me know so I can request a token for you as well. Ilya, you > > had > > > shown some interest in this as well, would you like me to include you? > > If > > > so which email address should I use? > > > > > > Anyone else who is planning to contribute an environment to the CI > > process > > > in the near to medium future? > > > > > > Cheers, > > > > > > *Will STEVENS* > > > Lead Developer > > > > > > *CloudOps* *| *Cloud Solutions Experts > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > > w cloudops.com *|* tw @CloudOps_ > > > > > > On Mon, Mar 7, 2016 at 6:36 PM, Erik Weber <terbol...@gmail.com> > wrote: > > > > > >> I guess the appropriate channel would be to create a jira ticket for > > >> INFRA: > > >> https://issues.apache.org/jira/browse/INFRA > > >> > > >> -- > > >> Erik > > >> > > >> On Mon, Mar 7, 2016 at 11:18 PM, Will Stevens <wstev...@cloudops.com> > > >> wrote: > > >> > > >> > This is kind of what I was expecting. Do you know who I would be > > >> > contacting? The permissions required are VERY minimal AND they have > > >> > already given the 'TravisCI' application the same permissions as we > > need > > >> > for this. > > >> > > > >> > How did we get the TravisCI application enabled and the permissions > > >> > accepted for that integration? > > >> > > > >> > I have been trying to thinking of ways to potentially work the > system > > >> from > > >> > that side. The main issue I have with this approach is that it is > > not a > > >> > single application that we want to give permission to. Ideally, we > > >> would > > >> > give each individual/organization who is contributing a CI > environment > > >> > their own token. In that case, we would have to register the CI of > > each > > >> > party as their own CI integration. > > >> > > > >> > Here are some ideas I have as a 'work around' to the problem: > > >> > > > >> > 1) Register a single upr CI application with Github and have the > > apache > > >> > guys enable the integration. This will give the application a > single > > >> > access token. I can then compile upr with the access token embedded > > >> into > > >> > the binary. I don't like this approach and I feel we would probably > > be > > >> > violating some ToS somewhere. > > >> > > > >> > 2) Provide a web server implementation that each CI party can use to > > >> > register their own CI endpoint as a Github application integration. > > >> Then > > >> > we have the apache guys enable each of them (which will be a harder > > >> sell to > > >> > them), but then each CI party will get their own token and will be > > able > > >> to > > >> > post back as themselves. This is also nice because if someone is > not > > a > > >> > good community member and misbehaves, their integration can be > revoked > > >> > without it affecting everyone else who has a CI integration. > > >> > > > >> > 3) Provide a single web server implementation that is registered as > a > > >> > Github Application Integration. This implementation is then > approved > > by > > >> > the apache guys for the cloudstack repo. This web server > > implementation > > >> > (let's call it upr_server) keeps our one and only access token. I > > then > > >> > modify the upr command line tool to take a token that is provided by > > the > > >> > upr_server when a CI party registers on the upr_server website. The > > >> > upr command > > >> > will actually target the upr_server box when posting statuses, etc, > > >> which > > >> > will essentially proxy the calls on to Github using the token that > was > > >> > approved for the integration. > > >> > > > >> > I think that 3) is probably the cleanest solution and would reduce > our > > >> > chances of getting banned by someone for being too cheeky. It is a > > >> whole > > >> > lot of trouble for nothing, but if they are going to be a stick in > the > > >> mud > > >> > about this and not allow access tokens to anything other than > official > > >> > github supported integrations, then I will have to make that work... > > >> > > > >> > Ideas? Thoughts? Rants? :P > > >> > > > >> > *Will STEVENS* > > >> > Lead Developer > > >> > > > >> > *CloudOps* *| *Cloud Solutions Experts > > >> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > >> > w cloudops.com *|* tw @CloudOps_ > > >> > > > >> > On Mon, Mar 7, 2016 at 4:12 PM, Remi Bergsma < > > >> rberg...@schubergphilis.com> > > >> > wrote: > > >> > > > >> > > Hi Will, > > >> > > > > >> > > This is the main problem: there’s no one except Apache Infra with > > >> access > > >> > > to the Github CloudStack repo. Even committers have to push to > > Apache > > >> > git, > > >> > > which is mirrored to Github. We can’t close a PR, set a label, > > change > > >> a > > >> > > title or whatever basic operation. You can ask them for a token. > > When > > >> I > > >> > (as > > >> > > the release manager) asked for any more permission than an > anonymous > > >> user > > >> > > has it was kindly refused. I really hope you’ll have more luck > but I > > >> > > wouldn’t count on it. > > >> > > > > >> > > > > >> > > > > >> > > Regards, > > >> > > Remi > > >> > > > > >> > > > > >> > > On 07/03/16 19:10, "williamstev...@gmail.com on behalf of Will > > >> Stevens" > > >> > < > > >> > > williamstev...@gmail.com on behalf of wstev...@cloudops.com> > wrote: > > >> > > > > >> > > >The main thing we have to sort out with this type of integration > > (as > > >> it > > >> > is > > >> > > >today) is the distribution of access tokens with the correct > > >> permissions > > >> > > on > > >> > > >the apache/cloudstack github repo. The required permissions are > > very > > >> > > >limited, but I don't know if we have access to create new tokens. > > >> If we > > >> > > >don't then I will have to develop an application integration > > >> workaround > > >> > to > > >> > > >make it easier for the people with access to the > apache/cloudstack > > >> repo > > >> > to > > >> > > >give the people running CI integrations access to update statuses > > >> (like > > >> > > the > > >> > > >current travis integration). > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > >If you have questions or feedback, please don't be shy. > > >> > > > > > >> > > >*Will STEVENS* > > >> > > >Lead Developer > > >> > > > > > >> > > >*CloudOps* *| *Cloud Solutions Experts > > >> > > >420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > >> > > >w cloudops.com *|* tw @CloudOps_ > > >> > > > > > >> > > >On Mon, Mar 7, 2016 at 12:36 PM, Bharat Kumar < > > >> > > bharat.ku...@accelerite.com> > > >> > > >wrote: > > >> > > > > > >> > > >> Hi guys, > > >> > > >> > > >> > > >> I am also working on the similar reporting problem, here is > what > > i > > >> did > > >> > > >> > > >> > > >> link to the report > > https://github.com/bvbharatk/cloud-stack/pull/1 > > >> > > >> > > >> > > >> I am thinking this is good enough for now, I want to start > > posting > > >> the > > >> > > >> results on each pr as shown in the above link. > > >> > > >> please give me your comments or suggestions. > > >> > > >> > > >> > > >> Thanks, > > >> > > >> Bharat > > >> > > >> On 05-Mar-2016, at 7:02 PM, Will Stevens < > wstev...@cloudops.com > > >> > <mailto: > > >> > > >> wstev...@cloudops.com>> wrote: > > >> > > >> > > >> > > >> Daan > > >> > > >> > > >> > > >> Regarding the obligatory provider id. I agree, but I am still > > >> trying > > >> > to > > >> > > >> figure out the details. Creating distinct runs that have their > > own > > >> > > status > > >> > > >> is done by setting the 'context'. I think we would need to > have > > >> two > > >> > > pieces > > >> > > >> to this. A provider id and an environment id. > > >> > > >> > > >> > > >> So for example. Lets assume that my provider id is 'CloudOps' > > and > > >> I > > >> > > have > > >> > > >> two different environments, one for 'KVM' and one for 'Xen' > (for > > >> > > example). > > >> > > >> I would then the tool would produce the following two > independent > > >> CI > > >> > > >> statuses. > > >> > > >> 'CloudOps - KVM' : with a basic description of the environment. > > >> > > >> 'CloudOps - Xen' : again with a basic description of the env. > > >> > > >> ... and so on ... > > >> > > >> > > >> > > >> I am still sorting out the details here as well as making it > easy > > >> for > > >> > > us to > > >> > > >> integrate this into the apache/cloudstack repo with the access > we > > >> > > currently > > >> > > >> have. Adding this is 'no biggy' for me because I am building > > this > > >> > tool > > >> > > as > > >> > > >> we speak, and trying to tailor it to our (ACS) needs, so this > > type > > >> of > > >> > > >> feedback is perfect as it allows me to adapt the tool before I > > get > > >> too > > >> > > >> deep. > > >> > > >> > > >> > > >> *Will STEVENS* > > >> > > >> Lead Developer > > >> > > >> > > >> > > >> *CloudOps* *| *Cloud Solutions Experts > > >> > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > >> > > >> w cloudops.com<http://cloudops.com> *|* tw @CloudOps_ > > >> > > >> > > >> > > >> On Sat, Mar 5, 2016 at 4:17 AM, Daan Hoogland < > > >> > daan.hoogl...@gmail.com > > >> > > >> <mailto:daan.hoogl...@gmail.com>> > > >> > > >> wrote: > > >> > > >> > > >> > > >> Will > > >> > > >> > > >> > > >> Gret work, especially the thing you are showing in link [4], I > > >> would > > >> > > like > > >> > > >> to make an enhancement request and that is a obligatory > provider > > >> id. > > >> > > Only > > >> > > >> if it is no biggy for you! > > >> > > >> Several people may decide to do a XVM on ChildrensOS for > instance > > >> and > > >> > > so we > > >> > > >> may be aware of an obscurity that is different. If one fails > and > > >> the > > >> > > other > > >> > > >> succeeds it is easily identified. > > >> > > >> > > >> > > >> Ilya, > > >> > > >> > > >> > > >> I have been playing with go and it is a very nice language for > > >> such a > > >> > > >> simple script, though it wasn't exactly designed for it. So > don't > > >> read > > >> > > my > > >> > > >> comment/question as an objection. But we do have > > >> > > >> bash,python,c#,java,javascript,xslt,sql at least. That is not > > >> counting > > >> > > the > > >> > > >> build system and I am sure the hyperv has some extra windows > > >> specific > > >> > > >> stuff. > > >> > > >> To me it is inherent to the nature of across platform > > orchestration > > >> > and > > >> > > >> provisioning system so it is fine. It is something to consider. > > On > > >> the > > >> > > >> other hand when bringing in new tools we don't make the choice > > >> so.... > > >> > I > > >> > > am > > >> > > >> ranting, I guess. > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> On Sat, Mar 5, 2016 at 7:38 AM, ilya < > > ilya.mailing.li...@gmail.com > > >> > > <mailto: > > >> > > >> ilya.mailing.li...@gmail.com>> wrote: > > >> > > >> > > >> > > >> I see where Daan is coming from :) I thought this would be > 4th, > > >> not > > >> > > >> exactly 7ths. > > >> > > >> > > >> > > >> I'm not against golang by any means (if anything - its my next > > >> "go" to > > >> > > >> language these days). > > >> > > >> > > >> > > >> Things to consider: > > >> > > >> > > >> > > >> Would notify-pr support proxy? I've been thinking on ways of > > >> > > >> contributing test runs, there would have to be few things i'd > > need > > >> to > > >> > > do. > > >> > > >> > > >> > > >> 1) massage the log content - such that no environment details > are > > >> > > >> exposed, i can probably handle this with sed/awk.. > > >> > > >> > > >> > > >> 2) i'm behind multiple firewalls with no internet access. > > however, > > >> > some > > >> > > >> lab environments might have a proxy, so it would be nice to > have > > a > > >> > > >> support for it. > > >> > > >> > > >> > > >> Thanks > > >> > > >> ilya > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> On 3/4/16 6:56 AM, Will Stevens wrote: > > >> > > >> Yes, I have most of it already built and will be releasing it > > later > > >> > > >> today > > >> > > >> or over the weekend. The reason I chose Golang is because it > can > > >> be > > >> > > >> cross > > >> > > >> compiled to be run on any system and distributed as a single > > binary > > >> > > >> with > > >> > > >> no > > >> > > >> dependencies. This means that no one will have to worry about > > >> > building > > >> > > >> it > > >> > > >> or having to change their environment at all in order to use > it. > > >> I am > > >> > > >> trying to lower the barrier to entry and make it as easy as > > >> possible > > >> > > >> for > > >> > > >> people to contribute back their CI execution details. > > >> > > >> > > >> > > >> *Will STEVENS* > > >> > > >> Lead Developer > > >> > > >> > > >> > > >> *CloudOps* *| *Cloud Solutions Experts > > >> > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > > >> > > >> w cloudops.com<http://cloudops.com> *|* tw @CloudOps_ > > >> > > >> > > >> > > >> On Fri, Mar 4, 2016 at 8:53 AM, Daan Hoogland < > > >> > daan.hoogl...@gmail.com > > >> > > >> <mailto:daan.hoogl...@gmail.com> > > >> > > >> > > >> > > >> wrote: > > >> > > >> > > >> > > >> Will, Do you have an implementation of notify-pr? I am asking > as > > >> you > > >> > > >> specify it will be implemented in golang which seems odd. It is > > not > > >> > > >> amongst > > >> > > >> the 7 or so languages already in use. > > >> > > >> > > >> > > >> On Fri, Mar 4, 2016 at 1:54 AM, Will Stevens < > > >> > > >> williamstev...@gmail.com<mailto:williamstev...@gmail.com>> > > >> > > >> wrote: > > >> > > >> > > >> > > >> Hey Everyone, > > >> > > >> As I am sure most of you are aware, I have been focusing a lot > on > > >> > > >> ways > > >> > > >> to > > >> > > >> get CI integrated back into the community. > > >> > > >> > > >> > > >> Today I build a little POC to validate some ideas and get a > feel > > >> for > > >> > > >> a > > >> > > >> potential approach for getting CI integrated into the Github > pull > > >> > > >> request > > >> > > >> workflow. > > >> > > >> > > >> > > >> There are multiple individuals/companies focusing on CI right > now > > >> > > >> (which > > >> > > >> is a good thing), but there has not really been much discussion > > >> > > >> (that I > > >> > > >> am > > >> > > >> aware of) for how these different CI runs will push back > results > > to > > >> > > >> the > > >> > > >> community. I want to make sure that nobody's work on this > topic > > >> goes > > >> > > >> to > > >> > > >> waste, so my goal is to provide a simple and consistent way for > > >> > > >> everyone > > >> > > >> to > > >> > > >> push their results back to the community. > > >> > > >> > > >> > > >> Here is the basic idea (please give feedback): > > >> > > >> - A simple cross platform command line tool with zero > > dependencies > > >> > > >> will > > >> > > >> be > > >> > > >> provided (and open sourced) which will handle the submission of > > the > > >> > > >> CI > > >> > > >> results back to the community. It is written in Golang and is > > >> > > >> currently > > >> > > >> called 'notify_pr'. > > >> > > >> - At the end of a CI execution, the CI should automate the > > >> execution > > >> > > >> of > > >> > > >> this tool to handle updating the appropriate PR with the > results. > > >> > > >> > > >> > > >> Configuration can be done via the command line or through an > INI > > >> > > >> file. > > >> > > >> Here is an example of the configuration details. The commit is > > the > > >> > > >> commit that the CI just executed against. > > >> > > >> > > >> > > >> token = <your github token> > > >> > > >> owner = apache > > >> > > >> repo = cloudstack > > >> > > >> commit = c8443496d3fad78a4b1a48a0992ce545bde299e8 > > >> > > >> > > >> > > >> summary_file = <a text file summary of the run> > > >> > > >> full_detail_dir = <a directory structure to be recursively > > uploaded > > >> > > >> to > > >> > > >> object store> > > >> > > >> full_detail_files = <a comma separated list of files to upload > to > > >> > > >> object > > >> > > >> store> > > >> > > >> store_api = <swift or s3> > > >> > > >> store_endpoint = <url endpoint> > > >> > > >> store_identity = <keystone identity or aws access key> > > >> > > >> store_secret = <keystone password or aws secret key> > > >> > > >> > > >> > > >> I have not yet implemented the object storage endpoints, but I > > have > > >> > > >> code > > >> > > >> to do it from a different project, so I just need to add it. I > > >> will > > >> > > >> be > > >> > > >> able to host my CI output in a swift object store, but others > may > > >> > > >> need > > >> > > >> to > > >> > > >> use AWS S3. Maybe we can get sponsorship for this storage. We > > >> will > > >> > > >> only > > >> > > >> keep the logs for a window of like a week or so on the object > > store > > >> > > >> so > > >> > > >> the > > >> > > >> storage usage will not be ever growing. > > >> > > >> > > >> > > >> Basically, the tool takes the details of the repository you are > > >> > > >> validating > > >> > > >> a Pull Request for and the commit you are building. It also > > takes > > >> > > >> the > > >> > > >> files you would like to push to the pull request. The summary > > file > > >> > > >> will > > >> > > >> be > > >> > > >> shown as text in the pull request comment and the other files > > will > > >> be > > >> > > >> uploaded to an object store and will be publically available > for > > a > > >> > > >> period > > >> > > >> of time (probably about a week and then get cleaned up, details > > >> TBD). > > >> > > >> The > > >> > > >> files to be uploaded to object store could be either specified > as > > >> > > >> individual files, OR a target directory and all the files in > that > > >> > > >> directory > > >> > > >> will be recursively uploaded to the object store. > > >> > > >> > > >> > > >> When the tool is run, it will scan through all the open pull > > >> requests > > >> > > >> in > > >> > > >> the target repository and when it finds the pull request > > >> > > >> corresponding > > >> > > >> to > > >> > > >> the commit in question, it will post the details as a comment > to > > >> that > > >> > > >> pull > > >> > > >> request. This functionality is currently working (see the > > attached > > >> > > >> screenshot). I can change the formatting and such, this is > just > > an > > >> > > >> example. > > >> > > >> > > >> > > >> This is still a very rough concept that I have only worked on > > for a > > >> > > >> day, > > >> > > >> but hopefully you guys agree that it is a good start towards a > > >> > > >> consistent > > >> > > >> feedback mechanism for the community to take advantage of the > > >> > > >> different > > >> > > >> distributed CI installations. > > >> > > >> > > >> > > >> Please voice your feedback and concerns. I am sure I have not > > >> > > >> thought > > >> > > >> of > > >> > > >> everything and we may still want to make fundamental changes to > > the > > >> > > >> approach, but I think the concept is solid. > > >> > > >> > > >> > > >> Cheers, > > >> > > >> > > >> > > >> Will > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> -- > > >> > > >> Daan > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> -- > > >> > > >> Daan > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> DISCLAIMER > > >> > > >> ========== > > >> > > >> This e-mail may contain privileged and confidential information > > >> which > > >> > is > > >> > > >> the property of Accelerite, a Persistent Systems business. It > is > > >> > > intended > > >> > > >> only for the use of the individual or entity to which it is > > >> addressed. > > >> > > If > > >> > > >> you are not the intended recipient, you are not authorized to > > read, > > >> > > retain, > > >> > > >> copy, print, distribute or use this message. If you have > received > > >> this > > >> > > >> communication in error, please notify the sender and delete all > > >> copies > > >> > > of > > >> > > >> this message. Accelerite, a Persistent Systems business does > not > > >> > accept > > >> > > any > > >> > > >> liability for virus infected mails. > > >> > > >> > > >> > > > > >> > > > >> > > > > > > > > > > > > -- > Daan >
