Re: [CentOS-docs] Finished a prototype of GSoC project: Implement a new doc toolchain
On Thu, Jul 09, 2015 at 02:11:26AM +0530, kunaal jain wrote: On Thu, Jul 9, 2015 at 1:19 AM, Brian (bex) Exelbierd b...@pobox.com They have both a private git (with gerrit) and a github repo.A If you open a PR on github it is closed by a bot with instructions to submit the PR via gerrit to their private repo.A When the PR is resolved and merged in the private repo, the entire commit is pushed to github awarding the social status that comes from a commit. I am not suggesting we use gerrit. I agree. Main reason is because of two way sync of pagure and github. Once that done, user even doesn't need to be aware that pagure exists. I agree right now we are doing something like openstack which you described, but we dont want that.A [...] We use Pagure as a part of the toolchain. Pagure is an excellent git-centered forge created byA Pierre-Yves Chibon pin...@pingoured.fr. Currently Pagure is in its early stage and the APIs and web hooks are changing and improving. We are working closely with Pierre to better integrate Pagure into the toolchain. In the future we plan to sync complete pull request so that discussion can happen either on github or Paugre, and contributor doesna**t need to leave github in the whole process, which lowers the barrier of contributing a lot. Do you have a timeline for the full PR mirroring? I agree that using github will definitely lower the barrier to contribution. My experience with sync problems has shown that two-way is problematic.A Do you have a plan for that? Can we achieve our goals by having a one-way sync from github to pagure?A This would seem to simplify the problem and provide us the protection - unless I am missing something. That is our priority. There was a couple of roadblocks, which I think are now cleared. So we will be again working on full sync, and by this weekend we hope we can achieve this. Yes two way sync is very problematic. A mess of GitHub and pagure webhooks and API, listening to each other. Our aim is two way sync, if we fail, we can fall back to one way sync from github to pagure. This will still achieve our goal, but then maybe not every CentOS docs maintainer be open to idea of using Github. We had some sort of these ideas in a discussion. So as far as I know, currently, you are using web-hooks to do sync between pagure and github. Did you consider using the underlying requests repo of the project? In other words, for each project in pagure, there are 4 git repos: - the code of the project (the repo people clone/fork/work on) - the doc repo, for storing the project's documentation if that project wishes to - the tickets repo of the project (a JSON representation of every issue ever opened against the project) - the request repo of the project (a JSON representation of every pull-request and discussion about it opened against the project) All links are available at the bottom of the project's overview page. Note: tickets and requests repos are private (only people with commit to the main repo can clone them) So when using the requests repo, upon receiving a notification from github, you could create/update the appropriate JSON blob representing the pull-request and push it to the requests repo. From their pagure should update its database via a git hook to reflect the new/update the PR in its UI. Note: while interesting this is a part of the code that has been completely tested yet, so we might run into a few bugs (in this case I guess we would run into a few settings not being loaded correctly, nothing too problematic normally). Note2: I wonder if the PR model in Pagure could not be reworked so that it defines the address of the repo from which to pull the changes, rather than pointing to a pagure's project. This way, we could open a PR on pagure from a repo hosted elsewhere. So many ideas :) Pierre ___ CentOS-docs mailing list CentOS-docs@centos.org http://lists.centos.org/mailman/listinfo/centos-docs
Re: [CentOS-docs] Finished a prototype of GSoC project: Implement a new doc toolchain
Hi Pierre, Thank you very much! So as far as I know, currently, you are using web-hooks to do sync between pagure and github. Did you consider using the underlying requests repo of the project? Yes, I also think it is the best way to track PRs. GitHub PRs are equivalents of Pagure requests. And we can include more metadata in requests. In other words, for each project in pagure, there are 4 git repos: - the code of the project (the repo people clone/fork/work on) - the doc repo, for storing the project's documentation if that project wishes to - the tickets repo of the project (a JSON representation of every issue ever opened against the project) - the request repo of the project (a JSON representation of every pull-request and discussion about it opened against the project) All links are available at the bottom of the project's overview page. Note: tickets and requests repos are private (only people with commit to the main repo can clone them) Really helpful, thanks! It is more flexible than using API / web hooks. If we run a pagure instance locally, we just directly modify the ticket repo, commit the changes and all set ;) Love it! So when using the requests repo, upon receiving a notification from github, you could create/update the appropriate JSON blob representing the pull-request and push it to the requests repo. We keep the code repo synced using a same workflow (pull from github to get the changes, then push to pagure) now. I think it will also work well for ticket repo. From their pagure should update its database via a git hook to reflect the new/update the PR in its UI. Note: while interesting this is a part of the code that has been completely tested yet, so we might run into a few bugs (in this case I guess we would run into a few settings not being loaded correctly, nothing too problematic normally). Note2: I wonder if the PR model in Pagure could not be reworked so that it defines the address of the repo from which to pull the changes, rather than pointing to a pagure's project. This way, we could open a PR on pagure from a repo hosted elsewhere. If we have this feature, that'll be awesome! The major problem of using requests now is about creating requests. Because we have to create a pagure project for each user who open a PR on github (so pagure can pull the changes from those projects). Also, we need to create an account for each user. That’ll be difficult to manage. If we can pull changes directly from github, then all we need is a link :) We are now implementing a two-way sync between pagure and github (mainly comments, so users and staff can discuss and don’t need to leave their own platform). Two-way syncs are a little bit tricky, though ;) I think we may encounter some problems when dealing with high frequency changes. So many ideas :) Pierre ___ CentOS-docs mailing list CentOS-docs@centos.org mailto:CentOS-docs@centos.org http://lists.centos.org/mailman/listinfo/centos-docs http://lists.centos.org/mailman/listinfo/centos-docs Best regards, Lei___ CentOS-docs mailing list CentOS-docs@centos.org http://lists.centos.org/mailman/listinfo/centos-docs
Re: [CentOS-docs] Finished a prototype of GSoC project: Implement a new doc toolchain
On Thu, Jul 9, 2015 at 1:19 AM, Brian (bex) Exelbierd b...@pobox.com wrote: Hi Lei, Hi, I am Kunaal the other GSoC student working on this toolchain. On Jun 20, 2015, at 11:21 AM, Lei Yang yltt1234...@gmail.com wrote: I am sorry it has taken me so long to test this workflow. Thanks a lot! We need testers and input from community. Hi, Kunaal and I have completed a prototype of the first part of our GSoC project (implement a new doc toolchain). Our goal is to implement a doc toolchain for short how-to docs and lower the barrier for new comers to contribute. I am not trying to be dense, but I want to make sure I understand the goals here: This is the workflow of the prototype: 1. Contributor creates a new pull request on GitHub A fairly well understand platform with lots of potential users - cool. Github also has some social cachet. That's the aim. Exploit this social cachet. 2. A corresponding issue is automatically created on Pagure 3. Staff and contributor discuss on Pagure This part is a bit odd to me, today. I believe your goal is to make sure the discussion happens in a CentOS controlled space in case github ever becomes an issue. However, it seems odd that we don't just mirror the whole PR over (see below for where I read the paragraph where you said this was coming ...). Right now it is weird to have to visit one site to see the change and another to discuss it. Furthermore I feel like we are creating an educational challenge (which could be resolved with some great docs and a bot that posted in the PR on github) to get people to understand what is going on. I agree! That time pagure was not having such webhooks and API. We are working with pingou(pagure creator). Fortunately it has now. Yes that's what bot's aim was, to point out to the link. 4. When the pull request can be accepted, it is marked as “fixed”, and the pull request on github is automatically merged OK 5. New change is synced to Pagure (doc website will update according to Pagure repo, to be implemented later) While this makes sense, it seems like we are barely using github and I am not sure it is helping us in this equation completely. Can you help me understand what I am missing? Openstack does something sort of similar, but backwards from our workflow. They have both a private git (with gerrit) and a github repo. If you open a PR on github it is closed by a bot with instructions to submit the PR via gerrit to their private repo. When the PR is resolved and merged in the private repo, the entire commit is pushed to github awarding the social status that comes from a commit. I am not suggesting we use gerrit. I agree. Main reason is because of two way sync of pagure and github. Once that done, user even doesn't need to be aware that pagure exists. I agree right now we are doing something like openstack which you described, but we dont want that. We have made a prototype performing the steps above, and is now running ;) Next we plan to integrate a CI so the doc can be built in real time and ready for staff to review. Could you please give us some suggestions about the workflow above and about the toolchain? The original GSoC idea can be found here: http://wiki.centos.org/GSoC/2015/Ideas#docs-toolchain I like the CI. I'd love to have us consider including something like emender in a future release that could do CI on the PR and post a response with any automated integration failures. Thanks! We don't have much knowledge about that, once we get there, maybe you can help us. I made a sample CI for listening to github repo changes, and automatically generating new site and deploying it. https://github.com/kunaaljain/test-centos-docs We use Pagure as a part of the toolchain. Pagure is an excellent git-centered forge created by Pierre-Yves Chibon pin...@pingoured.fr. Currently Pagure is in its early stage and the APIs and web hooks are changing and improving. We are working closely with Pierre to better integrate Pagure into the toolchain. In the future we plan to sync complete pull request so that discussion can happen either on github or Paugre, and contributor doesn’t need to leave github in the whole process, which lowers the barrier of contributing a lot. Do you have a timeline for the full PR mirroring? I agree that using github will definitely lower the barrier to contribution. My experience with sync problems has shown that two-way is problematic. Do you have a plan for that? Can we achieve our goals by having a one-way sync from github to pagure? This would seem to simplify the problem and provide us the protection - unless I am missing something. That is our priority. There was a couple of roadblocks, which I think are now cleared. So we will be again working on full sync, and by this weekend we hope we can achieve this. Yes two way sync is very problematic. A mess of GitHub and pagure webhooks and API, listening
Re: [CentOS-docs] Finished a prototype of GSoC project: Implement a new doc toolchain
Hi Lei, On Jun 20, 2015, at 11:21 AM, Lei Yang yltt1234...@gmail.com wrote: I am sorry it has taken me so long to test this workflow. Hi, Kunaal and I have completed a prototype of the first part of our GSoC project (implement a new doc toolchain). Our goal is to implement a doc toolchain for short how-to docs and lower the barrier for new comers to contribute. I am not trying to be dense, but I want to make sure I understand the goals here: This is the workflow of the prototype: 1. Contributor creates a new pull request on GitHub A fairly well understand platform with lots of potential users - cool. Github also has some social cachet. 2. A corresponding issue is automatically created on Pagure 3. Staff and contributor discuss on Pagure This part is a bit odd to me, today. I believe your goal is to make sure the discussion happens in a CentOS controlled space in case github ever becomes an issue. However, it seems odd that we don't just mirror the whole PR over (see below for where I read the paragraph where you said this was coming ...). Right now it is weird to have to visit one site to see the change and another to discuss it. Furthermore I feel like we are creating an educational challenge (which could be resolved with some great docs and a bot that posted in the PR on github) to get people to understand what is going on. 4. When the pull request can be accepted, it is marked as “fixed”, and the pull request on github is automatically merged OK 5. New change is synced to Pagure (doc website will update according to Pagure repo, to be implemented later) While this makes sense, it seems like we are barely using github and I am not sure it is helping us in this equation completely. Can you help me understand what I am missing? Openstack does something sort of similar, but backwards from our workflow. They have both a private git (with gerrit) and a github repo. If you open a PR on github it is closed by a bot with instructions to submit the PR via gerrit to their private repo. When the PR is resolved and merged in the private repo, the entire commit is pushed to github awarding the social status that comes from a commit. I am not suggesting we use gerrit. We have made a prototype performing the steps above, and is now running ;) Next we plan to integrate a CI so the doc can be built in real time and ready for staff to review. Could you please give us some suggestions about the workflow above and about the toolchain? The original GSoC idea can be found here: http://wiki.centos.org/GSoC/2015/Ideas#docs-toolchain http://wiki.centos.org/GSoC/2015/Ideas#docs-toolchain I like the CI. I'd love to have us consider including something like emender in a future release that could do CI on the PR and post a response with any automated integration failures. We use Pagure as a part of the toolchain. Pagure is an excellent git-centered forge created by Pierre-Yves Chibon pin...@pingoured.fr mailto:pin...@pingoured.fr. Currently Pagure is in its early stage and the APIs and web hooks are changing and improving. We are working closely with Pierre to better integrate Pagure into the toolchain. In the future we plan to sync complete pull request so that discussion can happen either on github or Paugre, and contributor doesn’t need to leave github in the whole process, which lowers the barrier of contributing a lot. Do you have a timeline for the full PR mirroring? I agree that using github will definitely lower the barrier to contribution. My experience with sync problems has shown that two-way is problematic. Do you have a plan for that? Can we achieve our goals by having a one-way sync from github to pagure? This would seem to simplify the problem and provide us the protection - unless I am missing something. To test it, you may create a PR on github. A corresponding issue will be created automatically on Pagure, so you may go to Pagure.io to check it. If you want to merge the PR, change the status of issue from Open to Fixed on Pagure (need admin access to the repo, if you want to test this part, please email me or Kunaal kunaa...@gmail.com mailto:kunaa...@gmail.com so we can add you as admin). Once the issue is marked as Fixed, the PR on github will be merged. Feel free to merge or reject my PR as you see fit :) Also, we are not very sure whether running a Pagure instance will add much workload to sys admin team. What’s your idea? I can't speak to the overhead, but I can say that I believe having Pagure running is a good idea. regards, bex___ CentOS-docs mailing list CentOS-docs@centos.org http://lists.centos.org/mailman/listinfo/centos-docs
