[OpenStack-Infra] Retiring our puppet modules
Hello everyone, For a while now we've been slowly moving towards a system that is deployed with less puppet and more ansible+docker. One of the side effects of that is we are no longer effectively maintaining our puppet modules. To make this more clear to people that may be using them, I am writing this email and we are also removing testing from repos that we no longer consume so that they can be formally retired. The first step is job removal which is beginning to happen in this change, https://review.opendev.org/#/c/720900/. The retirement process basically has us update the repo with a commit that removes content and replaces it with a README indicating the repo is retired. This means the process is reversible, the retirement commit can be reverted if necessary. If you are consuming these repos the first step you should take is to checkout the current HEAD commits before any retirement commits land. Then if you intend to keep maintaining the repos let us know and we can probably help you set up forks of some sort to do that. Hopefully, this isn't a surprise as we've been headed in this direction for a while. I wanted to make sure this got communicated directly before we start making downstream visible changes though. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] OpenDev team meeting April 7, 2020
Hello, The weekly meeting has a change of venue and will be held in #opendev-meeting. Same time and day: 19:00UTC Tuesdays. Also, going forward I'll be sending the agenda to service-disc...@lists.opendev.org. This week's agenda can be found at: http://lists.opendev.org/pipermail/service-discuss/2020-April/00.html. Feel free to join the new mailing list as pop in for the meeting tomorrow, Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] New OpenDev Communication Channels
Hello All, Recently, we've transitioned to using #opendev on Freenode for our synchronous IRC communications. Since then we've spun up a new service-disc...@lists.opendev.org mailing list [1] where we'll plan changes to services, notify of meetings, answer questions about services and usage, and otherwise communicate about OpenDev. service-annou...@lists.opendev.org [2] will remain for important announcements. If you're interested in the developer infrastructure we run please join us on these mailing lists. We encourage every developer to at least subscribe to the service-announce[2] list. It will be very low traffic, and used only for important announcements with wide impact. We are also going to start having our weekly team meeting in #opendev-meeting. The time will continue to be 19:00 UTC Tuesdays. See you there, Clark [1] http://lists.opendev.org/cgi-bin/mailman/listinfo/service-discuss [2] http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Contribution to the OpenStack Infra.
On Tue, Mar 31, 2020, at 3:16 PM, Javier Romero wrote: > Hello Team, > > Hope your are well. My name is Javier, live in Argentina and work as a > cloud engineer. Have been working as a Linux sysadmin for the las 10 > years in an Internet service provider. > > > Would like to know if I can start contributing with the OpenStack > Infra. I have expierence with KVM virtualization and Docker containers. > > > > Thank you very much for your attention and sorry for the inconvenience. No thank you! it is not an inconvenience at all. I think we spoke briefly on IRC and my suggestion there was that you look into one of the current efforts that is happening, do some code review to see how pieces fit together and even volunteer to push some changes as you learn. An incomplete but hopefully useful list of activities happening currently: * Deploy Gerrit with Ansible and Docker (migration off of puppet) https://review.opendev.org/#/q/topic:gerrit-docker+status:open * Deploy jitsi meet with etherpad integration for collaborative conference calls and brainstorming (this uses Docker too) https://review.opendev.org/#/q/topic:meetpad+status:open * I think we've recently discovered that we may want to redeploy etherpad under docker for this as well so that we have an up to date installation under the opendev.org domain. https://review.opendev.org/#/c/716442/ * Add Fedora 31 image builds to nodepool so that Zuul jobs can run on Fedora 31 https://review.opendev.org/#/q/(topic:f31+OR+topic:fedora-31)+status:open * Execute our Ansible and Docker config management from Zuul in a CD fashion rather than via cron. https://review.opendev.org/#/q/topic:infra-prod-zuul+status:open I've tried to provide links to inflight changes for each of those efforts. You can review the individual changes there to get an idea for what each of these things involves. Also, I totally understand that this might be a lot to take in. If you pick something that looks interesting and have questions I am more than happy to answer them. Either here on the mailing list or on IRC (I am clarkb). Don't be afraid to ask questions. I have to ask many questions myself :) I hope this helps and let me know if it doesn't (I'll keep trying), Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for March 31, 2020
We will meet at 19:00 UTC March 31, 2020 in #openstack-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** https://review.opendev.org/#/c/710057/ xwiki for wikis (this was rebased to address merge conflicts) * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates manage-projects seems to be running properly now splitting manage-projects out into zuul driven playbook *** Zuul as CD engine ** OpenDev *** Progress on Governance changes TC change and our docs update have landed. We should probably elect the project lead? Need to start formalizing the Advisory Board membership *** Location of Meetings and communication #opendev-meeting is being prepped and should be ready for meeting April 7, 2020 service-disc...@lists.opendev.org for asynchronous communication *** Making the new user experience a bit more pleasant "Getting Started" links on https://opendev.org that point to new Getting started document in infra-manual * General topics ** Trusty Upgrade Progress (clarkb 20200331) *** Wiki updates * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for March 24, 2020
We will meet tomorrow, March 24, at 19:00 UTC in #openstack-meeting with this agenda: == Agenda for next meeting == * Announcements ** http://lists.openstack.org/pipermail/foundation/2020-March/002854.html PTG going virtual * Actions from last meeting * Specs approval ** https://review.opendev.org/#/c/710057/ xwiki for wikis ** https://review.opendev.org/714189 meetpad * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Gerrit on docker updates *** Zuul as CD engine ** OpenDev *** Progress on Governance changes TC change and our docs update have landed. We should probably elect the project lead? Need to start formalizing the Advisory Board membership *** Gitea restarts should be done in proper order preventing Gerrit replication from losing events. We should try and confirm this after a gitea image update. * General topics ** Trusty Upgrade Progress (clarkb 20200324) *** Wiki updates ** Future location of this meeting (clarkb 20200324) * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for March 17, 2020
Sorry for the delay in getting this out. I had a power outage that kept my electronics off. We will meet with this agenda tomorrow, March 17 at 19:00UTC in #openstack-meeting: == Agenda for next meeting == * Announcements ** Switching to #opendev now - and merge https://review.opendev.org/#/c/711106/ ** http://lists.openstack.org/pipermail/foundation/2020-March/002852.html OSF email on 2020 events * Actions from last meeting * Specs approval ** https://review.opendev.org/#/c/710057/ xwiki for wikis * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Progress on Governance changes http://lists.openstack.org/pipermail/openstack-infra/2020-March/006603.html OpenDev as OSF pilot project https://review.opendev.org/#/c/710020/ Split OpenDev out of OpenStack governance. This change merged. https://review.opendev.org/#/c/703488 Updates to our project documentation with governance info. Now we need to land this change. * General topics ** Trusty Upgrade Progress (clarkb 20200317) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20200317) *** Now down to cleanup of servers ** nb01.opendev.org adventures (clarkb 20200317) *** Podman and docker have different mount propagation rules. Switching back to docker for consistency of behavior. *** Will be redeployed as nb04.opendev.org to avoid hostname conflicts (even if not strictly required anymore). * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Team Meeting Agenda for March 10, 2020
We will meet in #openstack-meeting at 19:00 UTC (note the DST change in some parts of the world) March 10, 2020 with this agenda: == Agenda for next meeting == * Announcements ** http://lists.openstack.org/pipermail/foundation/2020-March/002852.html OSF email on 2020 events ** DST change has happened in some parts of the world. Double check your calendar entries. * Actions from last meeting * Specs approval ** https://review.opendev.org/#/c/710057/ xwiki for wikis * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Progress on Governance changes http://lists.openstack.org/pipermail/openstack-infra/2020-March/006603.html OpenDev as OSF pilot project https://review.opendev.org/#/c/710020/ Split OpenDev out of OpenStack governance. https://review.opendev.org/#/c/703488 Updates to our project documentation with governance info * General topics ** Trusty Upgrade Progress (clarkb 20200310) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20200310) *** Now down to cleanup of servers ** Discuss future IRC channel usage (frickler 20200310) *** Splitting the channels has begun. Openstack events removed from #opendev. *** https://review.opendev.org/#/c/711106/1 if we get quorum on that change we can announce the migration then start holding people to it after an agreed on date. ** FortNebula now OpenEdge Cloud (clarkb 20200310) *** Has been redeployed and is being spun back up again. * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for March 3, 2020
We will meet tomorrow, March 3, at 19:00UTC in #openstack-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** https://review.opendev.org/#/c/709579/ Cleaning up python dev tools on our CI images. ** https://review.opendev.org/#/c/709236/ Website activity stats ** https://review.opendev.org/#/c/710057/ xwiki for wikis * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Progress on Governance changes http://lists.openstack.org/pipermail/openstack-infra/2020-March/006603.html OpenDev as OSF pilot project https://review.opendev.org/#/c/710020/ Split OpenDev out of OpenStack governance. https://review.opendev.org/#/c/703488 Updates to our project documentation with governance info * General topics ** Trusty Upgrade Progress (clarkb 20200303) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20200303) *** static.openstack.org transition should be complete including redirects. ** Discuss future IRC channel usage (frickler 20200226) *** We have both #openstack-infra and #opendev, maybe after the governance change we can now focus on using the latter? *** In particular the high number of duplicated gerritbot msgs is seen as an annoyance, some are even seen yet once more elsewhere, like for zuul/zuul. * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] OpenDev infra as an OSF pilot project
On Sun, Mar 1, 2020, at 11:20 AM, Jonathan Bryce wrote: > Hi everyone, > > I saw some of the discussions on different channels last week about the > ongoing move of the OpenDev infra services out of OpenStack project and > TC governance. One of the questions that was raised was around setting > it up as an OSF pilot project. I wanted to send an email to this list > to see if that was something the team was interested in moving forward > on. > > As a pilot project, it would create some official standing for the new > effort that would make it clear it’s something that is still supported > by the OSF community and staff. It would also provide additional > opportunities for education and exposure as part of the foundation’s > overall activities. While the OpenDev infra services are somewhat > different than the other projects we have piloted (e.g. Zuul), I think > the process would still work and could be helpful to complete the > transition to a more standalone community both from a governance and > perception standpoint. > > Pilot projects are initiated through action of the foundation staff and > over time may be confirmed by the Board as a top-level project with > long-term support. I personally would be supportive of taking the pilot > step, and would like to hear thoughts from those of you who are > directly engaged in it. I'm in favor of it. I think my biggest concern is that it could be awkward to sort through the confirmation process. Perhaps you could elaborate on how you think that might work given the current framework? Also, I'm not sure the entire infra team is familiar with this process so a bit more information on the process and what would be required of us would be useful. (I'd try but I'm sure to get it wrong). > > Thanks, > > Jonathan ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for February 25, 2020
We will meet at 19:00 UTC Tuesday, February 25 in #openstack-meeting with this agenda. See you there. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval ** https://review.opendev.org/#/c/709579/ Cleaning up python dev tools on our CI images. *** https://etherpad.openstack.org/p/pTFF4U9Klz : clarkb writeup of major issues *** https://review.opendev.org/707499 : discussion in comments *** https://review.opendev.org/707513 : use venv for glean *** https://review.opendev.org/707750 : use venv in project-config elements; drop pip-and-virtualenv inclusion from element and move to individual configs; add node type with no pip-and-virtualenv. this can be used for job testing. ** https://review.opendev.org/#/c/709236/ Website activity stats * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Progress on Governance changes https://review.opendev.org/#/c/703134/ Split OpenDev out of OpenStack governance. https://review.opendev.org/#/c/703488 Updates to our project documentation with governance info * General topics ** Xwiki for opendev (ttx 20200224) ** Vancouver PTG attendance. (clarkb 20200225) *** Request space for a small group in Vancouver. Mentioned that we can be flexible, but having some official time is useful so that people can find us with questions. Then with a small group it is easy enough to meet in an ad hoc fashion (this worked well in Shanghai). ** Trusty Upgrade Progress (clarkb 20200225) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20200225) *** static.openstack.org transition should be complete *** still tasks remaining (haproxy redirects, few other publishing sites) * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for February 18, 2020
We will meet at 19:00 UTC February 18, 2020 in #openstack-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Progress on Governance changes https://review.opendev.org/#/c/703134/ Split OpenDev out of OpenStack governance. https://review.opendev.org/#/c/703488 Updates to our project documentation with governance info *** Upgrade gitea to 1.10.3 https://review.opendev.org/#/c/705804/ * General topics ** Vancouver PTG attendance. (clarkb 20200218) *** That time of year where we need to decide if we are going and if so how much time and space we need. ** Trusty Upgrade Progress (clarkb 20200218) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20200218) *** static.openstack.org transition should be complete *** still tasks remaining (haproxy redirects, few other publishing sites) ** New Airship CI cloud (clarkb 20200218) ** pip-and-virtualenv removal (ianw 20200218) *** https://etherpad.openstack.org/p/pTFF4U9Klz : clarkb writeup of major issues *** possible removal plan: use py3 venv (built in) on all platforms for build-time "utility" installs (glean, bindep, stackviz, whatever); ship images without any non-distro-packaged tools; add for backwards compatibility via base jobs *** https://review.opendev.org/707499 : discussion in comments *** https://review.opendev.org/707513 : use venv for glean *** https://review.opendev.org/707750 : use venv in project-config elements; drop pip-and-virtualenv inclusion from element and move to individual configs; add node type with no pip-and-virtualenv. this can be used for job testing. * Open discussion ** Need to clarify logging requirements from https://docs.openstack.org/infra/system-config/third_party.html -- mainly the "logs must be browsable; logs requiring download, installation or login to access are not acceptable" phrase. (ssbarnea 20200217) *** Does it apply to any logs or only tempest ones? Current indentation seems wrong, was this expected to be top level? *** Is ever accepted to produce combined .tar.gz files with host-logs? *** In which conditions it would be acceptable to archive a big log file? Example publishing a 1GB single log file is clearly not of any use for browsing it. I suspect after certain size is ok to archive them but this is not mentioned. What would be the magic threshold? ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for February 11, 2020
We will meet on February 11, 2020 at 19:00 UTC in #openstack-meeting with this agenda. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Progress on Governance changes https://review.opendev.org/#/c/703134/ Split OpenDev out of OpenStack governance. https://review.opendev.org/#/c/703488 Updates to our project documentation with governance info *** Possible gitea/go-git bug in current version of gitea we are running https://storyboard.openstack.org/#!/story/2006849 *** Gitea has added a commit cache and closed the slow performance bug. We should consider an upgrade as soon as is reasonable. * General topics ** Vancouver PTG attendance. (clarkb 20200211) *** That time of year where we need to decide if we are going and if so how much time and space we need. ** Trusty Upgrade Progress (clarkb 20200211) *** Status.o.o mostly done. Needs new gerritlib release for e-r IRC bot to get Gerrit events. *** Wiki updates *** Refstack needs changes to upstream code base to make it deployable on our docker python-base image. ** static.openstack.org (ianw,corvus,mnaser,fungi 20200211) *** Publishing updates ** New Arm64 cloud (ianw 20200211) *** nb03 unable to talk to https://us.linaro.cloud:5000 (still) ** New Airship CI cloud (clarkb 20200211) * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for February 4, 2020
We will meet at 19:00UTC on February 4, 2020 in #openstack-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting ** Clarkb out Wednesday February 5, 2020. * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Progress on Governance changes https://review.opendev.org/#/c/703134/ Split OpenDev out of OpenStack governance. https://review.opendev.org/#/c/703488 Updates to our project documentation with governance info *** Possible gitea/go-git bug in current version of gitea we are running https://storyboard.openstack.org/#!/story/2006849 *** Gitea has added a commit cache and closed the slow performance bug. We should consider an upgrade as soon as is reasonable. * General topics ** Trusty Upgrade Progress (clarkb 20200204) *** Status.o.o mostly done. Needs new gerritlib release for e-r IRC bot to get Gerrit events. *** Wiki updates *** Refstack getting docker'd due to stale puppet needing a rewrite anyway. Need changes in refstack to land and make docker image buildable. ** static.openstack.org (ianw,corvus,mnaser,fungi 20200204) *** Publishing updates ** New Arm64 cloud (ianw 20200204) *** nb03 unable to talk to https://us.linaro.cloud:5000 (still) ** New Airship CI cloud (clarkb 20200204) * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Help / Pointers on mirroring https://opendev.org/starlingx in github.com
On Thu, Jan 30, 2020, at 5:21 AM, Waines, Greg wrote: > > Hello, > > > I am working in the OpenStack StarlingX team. > > We are working on getting StarlingX certified through the CNCF > conformance program, > https://www.cncf.io/certification/software-conformance/ . > > > ( in the same way that the OpenStack Magnum project got certified with > CNCF ... you can search for them on the above page ) > > > In order for the logo to be shown as based on open-source, CNCF > requires that the code be mirrored on github.com . > > > e.g. OpenStack Magnum has done this: https://github.com/openstack/magnum > > In fact, not sure why, but almost all the core OpenStack projects have > github mirrors: nova, neutron, keystone, cinder, glance, etc. > > > Since so many of the OpenStack projects currently mirror to github.com, > I was hoping there was a well-documented recipe for setting this up. > > > **??? Is there a recipe for mirroring an openstack project’s opendev > repositories in github.com ???** > > **Or someone that can help out in doing this ?** Roman provided good feedback[0] on the thread you started on openstack-discuss. I suggest we keep further discussion there as that serves as a good starting point. Note that magnum uses the legacy replication tooling, but Airship uses the modern process that Roman describes (this is what should be used). [0] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012268.html > > > Let me know, > > Greg. > > > > p.s. I cc’d Thierry because I noticed that there are repos for all the > StarlingX subprojects in github.com now > > e.g. https://github.com/openstack/stx-metal > > But with a final commit/comment from Thierry that these repos have been > retired and moved to opendev ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for January 28, 2020
We will meet in #openstack-meeting at 19:00 UTC January 28, 2020 with this agenda. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Progress on Governance changes https://review.opendev.org/#/c/703134/ Split OpenDev out of OpenStack governance. https://review.opendev.org/#/c/703488 Updates to our project documentation with governance info *** Possible gitea/go-git bug in current version of gitea we are running https://storyboard.openstack.org/#!/story/2006849 * General topics ** Trusty Upgrade Progress (clarkb 20200128) *** status.o.o in progress. Need gerritlib updates and release. *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20200128) ** New Arm64 cloud (clarkb 20200128) *** nb03 unable to talk to https://us.linaro.cloud:5000 ** Airship CI cloud meeting Wednesday at 1600UTC (clarkb 20200128) * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Touching base; Airship CI cluster
On Tue, Jan 7, 2020, at 1:45 AM, Roman Gorshunov wrote: > Hello Clark, > > Thank you for your reply. Meeting time is OK for me. I have forwarded > invitation to Pete Birley and Matt McEuen, they would hopefully join > us. I wanted to make sure we got a summary of this meeting sent out. Notes were kept at https://etherpad.openstack.org/p/Wqxwce1UDq Airship needs to test their All in One deployment tool. This tool deploys their entire bootstrapping system into a single VM which is then used to deploy other software which may be clustered. Because the production usage of this tool is via a VM it is actually important to be able to test the contents of that VM in CI and that is what creates the memory requirements for Airship CI. We explained the benefits of being able to run Airship CI on less special hardware. Airship gains redundancy as more than one provider can supply these resources, reliability should be improved as nested virt has been known to be flaky, and better familiarity within the community with global resources means that debugging and working together is easier. However, we recognize that Airship has specific constraints today that require more specialized testing. The proposed plan is to move forward with adding a new cloud, but have it provide specialized and generic resources. The intent is to address Airship's needs of today with the expectation that they will work towards running on the generic resources. Having generic resources ensures that the infra team has exposure to this new cloud outside the context of Airship. This improves familiarity and debuggability of the system. It is also more equitable as other donations are globally used. Also, Nodepool doesn't actually allow us to prevent consumption of resources exposed to the system; however, we would ask that specialized resources only be used when necessary to test specific cases as with Airship. This is similar to our existing high memory, multi numa node, nested virt enabled test flavors. For next steps we'll work to add the new cloud with the two sets of flavors, and Airship will begin investigating what a modified test setup looks like to run on our generic resources. We'll see where that takes us. Let me know if this summary needs editing or updating. Finally, we'll be meeting again Wednesday January 29, 2020 at 1600UTC to followup on any questions now that things should be moving. I recently used jitsi meet and it worked really well so want to give that a try for this. Lets meet at https://meet.jit.si/AirshipCICloudFun. Fungi says you can click the circled "i" icon at that url to get dial in info if necessary. If for some reason jitsi doesn't work we'll fall back to the method used last time: https://wiki.openstack.org/wiki/Infrastructure/Conferencing room 6001. See you there, Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for January 21, 2020
We will meet in #openstack-meeting on January 21, 2020 at 19:00UTC with this agenda. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Progress on Governance changes https://review.opendev.org/#/c/703134/ Split OpenDev out of OpenStack governance. https://review.opendev.org/#/c/703488 Updates to our project documentation with governance info *** Possible gitea/go-git bug in current version of gitea we are running https://storyboard.openstack.org/#!/story/2006849 * General topics ** Trusty Upgrade Progress (clarkb 20200107) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20200107) *** Need reviews on https://review.opendev.org/#/q/status:open+topic:static.opendev.org ** New Arm64 cloud * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Fwd: CVE References in LPs are messed up after centos feature branch rebase
On Tue, Jan 14, 2020, at 2:46 PM, Saul Wold wrote: > > > On 1/14/20 2:08 PM, Clark Boylan wrote: > > On Tue, Jan 14, 2020, at 1:49 PM, Saul Wold wrote: > >> Clark: > >> > >> Happy New Year! > >> > >> Circling back on this after the holidays, where do we stand, I am about > >> to do another merge for the feature branch, since for the work we are > >> doing across repos is best via feature branch vs Depends-On as talked > >> about below. > >> > >> We talked about only reporting an abbreviated commit message when doing > >> a merge commit, and I asked if a storyboard or launchpad was needed, did > >> I lose a reply? > > > > I don't think so, it has been quiet. You can file a bug if you like, but I > > expect that this is one of those cases where the quickest path to > > resolution will be for someone on the StarlingX side to push up a change > > with the edits that were suggested. > > OK, I guess that falls to me! I need to set up appropriate testing, > unless there is some existing test area I can play with. We do have a review-dev.openstack.org we can deploy it to. I believe that server is setup to comment on launchpad for at least one of its projects? Monty would probably know since he has been converting our Gerrit config management recently. If you'd prefer to do it locally all Gerrit does on those events is call the hook scripts with a specific set of parameters. Those parameters are documented here: https://gerrit.googlesource.com/plugins/hooks/+/refs/heads/stable-2.13/src/main/resources/Documentation/hooks.md#change_merged. Then you can execute the script locally as if you were Gerrit. This is probably reasonably straightforward though you will want to set up a throw away LP bug to edit. > > Sau! ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Fwd: CVE References in LPs are messed up after centos feature branch rebase
On Tue, Jan 14, 2020, at 1:49 PM, Saul Wold wrote: > Clark: > > Happy New Year! > > Circling back on this after the holidays, where do we stand, I am about > to do another merge for the feature branch, since for the work we are > doing across repos is best via feature branch vs Depends-On as talked > about below. > > We talked about only reporting an abbreviated commit message when doing > a merge commit, and I asked if a storyboard or launchpad was needed, did > I lose a reply? I don't think so, it has been quiet. You can file a bug if you like, but I expect that this is one of those cases where the quickest path to resolution will be for someone on the StarlingX side to push up a change with the edits that were suggested. > > Remember not all of us are on the Infra mailing list. > > Thanks >Sau! ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Infra Meeting agenda for January 14, 2020
We will meet tomorrow, January 14 at 19:00UTC in #openstack-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Foundation Individual Board Member Election happening now. Ends January 17 17:00UTC * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Gitea restarts for upgrading appear to be able to miss gerrit replication events. No errors logged on either side. *** Possible gitea/go-git bug in current version of gitea we are running https://storyboard.openstack.org/#!/story/2006849 * General topics ** Trusty Upgrade Progress (clarkb 20200107) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20200107) *** Need reviews on https://review.opendev.org/#/q/status:open+topic:static.opendev.org * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Infra Meeting Agenda for January 7, 2020
We will meet at 19:00UTC in #openstack-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Possible gitea/go-git bug in current version of gitea we are running https://storyboard.openstack.org/#!/story/2006849 *** Bug seems to remain present after Gitea 1.10 upgrade * General topics ** Trusty Upgrade Progress (clarkb 20200107) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20200107) *** Need reviews on https://review.opendev.org/#/q/status:open+topic:static.opendev.org ** Project renames (clarkb 20200107) * Open discussion Welcome back from the holidays! Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Touching base; Airship CI cluster
On Fri, Jan 3, 2020, at 8:29 AM, Gorshunov, Roman wrote: > Hello Christopher, Clark, OpenStack-Infra, > > Thank you for your help. > > Moving conversation to OpenStack-Infra mailing list. > > Answers: > OS images available are OK for us. We primarily target Ubuntu 18.04 as > a base image. > > > For sizing our current test VMs are 8vcpu, 8GB RAM, 80GB of disk. > This is the VM we would be running: 64 GB Ram, 200 GB block storage, 16 > VCpu, + nested virtualization > > > Is it possible that AirShip could run these test jobs on 3 > node multinode jobs using our existing nested-virt flavors? > No, we can’t. We really need to run nested VMs inside a very big VM to > simulate end-to-end baremetal deployments (with PXE booting, etc). > Nested VMs would be up to 16GB RAM. As mentioned in the earlier email nested virt is one of my big concerns with this as it hasn't been very stable for us in the past. Coupled with potentially having a single provider of CI resources this could lead to very flaky testing with no alternatives. Thinking out loud here, could you test the PXE provisioning independent of the configuration management and workload of the PXE provisioned resources? Then you could avoid nested virt when PXE booting and use qemu emulation which should be more reliable. I believe this is how Ironic does their testing. Then we can take advantage of our existing abilities to run multinode testing to check configuration management and workloads assuming the PXE boots succeeded (because this is tested separately). I expect this would give you more reliable testing over time as you avoid the nested virt problem entirely. This may also make the test environment fit into existing resources allowing you to take advantage of multi cloud availability should problems arise in any single test node provider cloud. > > > Airship could choose to accept the risk of having a single provider, but I > > would caution against this. > We currently have one provider, planning to have two providers soon. > > > The other policy thing we should consider is if these resources would > be available to the large pool of users or if they should be > considered airship specific. > We would prefer them to be Airship-specific, primarily because only we > would be running those huge VMs, and other providers do not have this > option at the moment for us to be able to utilize their hardware. > > Please, let’s set up a call for us at the time you would find convenient. Does 16:00UTC Thursday January 9, 2020 work for you? If so we should be able to use our asterisk server, https://wiki.openstack.org/wiki/Infrastructure/Conferencing, room 6001. Let me know if you cannot connect to that easily and we can set up a jitsi meet (https://meet.jit.si/) instead. > > -- > Roman Gorshunov > Principal, Systems Engineering > > From: Christopher Price > Sent: Thursday, December 12, 2019 11:00 AM > To: Gorshunov, Roman > Cc: paye...@gmail.com > Subject: Re: Touching base > > Hi Roman, > > I have been in touch with a few people and it seems there is a way to > solve this across the groups, however there is some thinking and effort > involved in making it happen. > We should set up a call with the AirShip and some of the infra team to > discuss and align on objectives, however I think first it’s worth > having a conversation in the AirShip community to understand what is > needed. Here is a basic rundown from Clarke, sorry for the long mail > with Q included: > > On 12/6/19 4:55 AM, Christopher Price wrote: > > Hey Clark, > > > > Reaching out to chat about getting some of the more unique AirShip gating > > use-cases set up in openinfra. > > > > Some text below in my question to Thierry for context, but to summarize: > > - Airship needs to do "huge VM" based testing with nested virtualization > > enabled for gating in opendev > > - A few companies are wiling to put up some hardware to support that - > > maintenance and support from the companies > > - I'd like to bring the "what does infra expect" topic forward to see what > > needs to happen and what we need to ensure to get the gating in place for > > these items > > > > I think the plan is to use something like airskiff or airship-in-a-bottle > > as a method of doing deploy testing for gating. > > Requires a Ubuntu 16.04 VM (minimum 4vCPU/20GB RAM/32GB disk) to run. > > > >https://urldefense.proofpoint.com/v2/url?u=https-3A__airship-2Dtreasuremap.readthedocs.io_en_latest_airskiff.html=DwMGaQ=LFYZ-o9_HUMeMTSQicvjIg=qhOWbb99YOfK35xi5gJyuCTk8aVUkJ-JqhgPQdVrAoo=R4VBrtcCr7N-f_2HymjTWkL3JH9SWSQGg3U_SZh32iM=4-zR8RIZpy1O2ZMeOjlqVklESP4M1GoUWP87iKuV2H8= > > > >https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_airshipit_airship-2Din-2Da-2Dbottle=DwMGaQ=LFYZ-o9_HUMeMTSQicvjIg=qhOWbb99YOfK35xi5gJyuCTk8aVUkJ-JqhgPQdVrAoo=R4VBrtcCr7N-f_2HymjTWkL3JH9SWSQGg3U_SZh32iM=KfFmZvKHEh4LRqnDhAYol88lMLbBtGnQE_mKjUz8QTI= > > We currently build images
Re: [OpenStack-Infra] Fwd: CVE References in LPs are messed up after centos feature branch rebase
On Tue, Dec 17, 2019, at 10:16 AM, Saul Wold wrote: > > > On 12/16/19 8:22 AM, Clark Boylan wrote: > > On Mon, Dec 16, 2019, at 7:46 AM, Saul Wold wrote: > >> > >> Hi Clark, > >> > >> Sorry, I only get the archive of Infra and Ghada is not on the list, if > >> you can please reply to us and the list that would be great. > >>> > >>> I think what happened here is you merged bug fixes (in this case cve bug > >>> fixes) from master into a feature branch. Then when you pushed that merge > >>> commit and merged it, the bot noticed that those bug fixes had merged to > >>> the feature branch and commented with those details on the bug. I believe > >>> this is "correct" behavior from the bot. > >> > >> Is there a different way to do the merge activity? > >> > >>> > >>> Is the issue the existence of comments like > >>> https://bugs.launchpad.net/starlingx/+bug/1844579/comments/18 on the > >>> bugs? Or is there some other metadata that is being added that I am > >>> missing? > >>> > >> Yes, that comment does not belong with that bug and because the comment > >> includes CVE-2019-X formating it adds the CVE References metadata also. > > > > Can you expand on this? Why does the comment not belong with the bug? The > > bug was fixed on the f/centos8 branch and that is what the comment is > > telling you. Where is the CVE References metadata? > > > The "merge commit" message contains all the commits that are part of the > merge commit. I guess the hook sees the merge commit with the Closes: > tag and adds the complete commit message to the associated launchpad > bugs (in the case of multiple closes due to multiple commit messages in > the merge commit. > > Since that larger "merge commit" message contains CVE reference they get > added to the Closes: tagged bugs. Look again at > https://bugs.launchpad.net/starlingx/+bug/1844579 > Below the description is the CVE Reference with links to the CVE > mentioned. This launchpad has nothing to do with the CVEs in question. > I guess this is done inside launchpad, not in the opendev bugtask. > > Does that make more sense? Yes, that helps. And yes I believe launchpad is doing its own string scraping and deciding to list those CVEs. I don't believe we are triggering that explicitly. > > >> > >>> If we don't want comments like that to appear you'd need to modify your > >>> merged trees so that bug fixes don't go from master into the feature > >>> branch. Or we'd need to come up with some rule set we can apply to the > >>> bot to filter bugs out in certain circumstances. > >> > >> Modifying the merge trees would defeat the purpose of doing the merge I > >> think. Does this issue not affect other projects or are we yet again > >> doing strange operations in StarlingX ;-)! Not sure how hare it would > >> be to filter for feature branches. > > > > Yes, you probably don't want to change the merge trees as the idea here is > > to bring the feature branch up to date, and probably the most important > > aspect of that is ensuring you've merged security fixes. > > > > Use of feature branches at all may qualify as "strange". Most projects tend > > to develop against their target branch. You'll see large change series from > > nova for example rather than creating feature branches for that work. This > > means most projects are never in a situation to potentially hit this > > problem. One major historical exception to this has been the swift project. > > It is possible they have run into this problem but ignored it? Or not seen > > it as problematic? > > > I think we chose to use feature branches since there are multiple repos > in StarlingX and we need a way to coordinate work across them. Note, Zuul's depends-on functionality is designed to address the need for coordinating work between repos without needing to drastically change workflow. > > They might not have as many CVE reference also, since StarlingX has many > references to Linux Userspace which can contain more CVEs. > > > I did double check that the change merge hook code doesn't handle feature > > branches as a special case already (openstack uses the feature/ prefix not > > f/ so thought maybe there was a difference in matchers?) but I found > > nothing. > > https://opendev.org/opendev/jeepyb/src/branch/master/jeepyb/cmd/update_bug.py#L222-L252 > > is the code in question and what
[OpenStack-Infra] Meeting Agenda for December 17, 2019
We will meet for the last time in 2019 on December 17 at 19:00UTC in #openstack-meeting with this agenda: == Agenda for next meeting == * Announcements ** No meeting December 24, 2019 and December 31, 2019 * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Possible gitea/go-git bug in current version of gitea we are running https://storyboard.openstack.org/#!/story/2006849 *** Bug seems to remain present after Gitea 1.10 upgrade * General topics ** Trusty Upgrade Progress (clarkb 20191217) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20191217) *** Infra-root needs to create AFS volumes. ** Discussion on retiring services that are largely unmaintained or unused (clarkb 20191217) *** Want to start this discussion as services like Ask are barely on life support and we should probably clean them up. *** https://etherpad.openstack.org/infra-service-list * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Fwd: CVE References in LPs are messed up after centos feature branch rebase
On Mon, Dec 16, 2019, at 7:46 AM, Saul Wold wrote: > > Hi Clark, > > Sorry, I only get the archive of Infra and Ghada is not on the list, if > you can please reply to us and the list that would be great. > > > > I think what happened here is you merged bug fixes (in this case cve bug > > fixes) from master into a feature branch. Then when you pushed that merge > > commit and merged it, the bot noticed that those bug fixes had merged to > > the feature branch and commented with those details on the bug. I believe > > this is "correct" behavior from the bot. > > Is there a different way to do the merge activity? > > > > > Is the issue the existence of comments like > > https://bugs.launchpad.net/starlingx/+bug/1844579/comments/18 on the bugs? > > Or is there some other metadata that is being added that I am missing? > > > Yes, that comment does not belong with that bug and because the comment > includes CVE-2019-X formating it adds the CVE References metadata also. Can you expand on this? Why does the comment not belong with the bug? The bug was fixed on the f/centos8 branch and that is what the comment is telling you. Where is the CVE References metadata? > > > If we don't want comments like that to appear you'd need to modify your > > merged trees so that bug fixes don't go from master into the feature > > branch. Or we'd need to come up with some rule set we can apply to the bot > > to filter bugs out in certain circumstances. > > Modifying the merge trees would defeat the purpose of doing the merge I > think. Does this issue not affect other projects or are we yet again > doing strange operations in StarlingX ;-)! Not sure how hare it would > be to filter for feature branches. Yes, you probably don't want to change the merge trees as the idea here is to bring the feature branch up to date, and probably the most important aspect of that is ensuring you've merged security fixes. Use of feature branches at all may qualify as "strange". Most projects tend to develop against their target branch. You'll see large change series from nova for example rather than creating feature branches for that work. This means most projects are never in a situation to potentially hit this problem. One major historical exception to this has been the swift project. It is possible they have run into this problem but ignored it? Or not seen it as problematic? I did double check that the change merge hook code doesn't handle feature branches as a special case already (openstack uses the feature/ prefix not f/ so thought maybe there was a difference in matchers?) but I found nothing. https://opendev.org/opendev/jeepyb/src/branch/master/jeepyb/cmd/update_bug.py#L222-L252 is the code in question and what we'd end up updating if we wanted to apply some rule set to the bot around feature branches. Something else to keep in mind, there has been some discussion of replacing these existing bots with Zuul jobs similar to how github replication is done. That could possibly give different repos far more flexibility through Zuul configuration specific to that repo. This may be another approach worth taking if we find we end up doing something StarlingX specific. > > Thanks > Sau! > > > > > > On 12/13/19 8:48 AM, Saul Wold wrote: > > > > Hello Infra team: > > > > Apparently something got messed up with Launchpad and updating a number > > of starlingx repos with a feature branch. > > > > I was following the methodology of updating a feature branch with > > changes from master via merges and I guess when I pushed that to gerrit > > and it merged, it caused some Launchpad ugliness. See email below. > > > > Thoughts? > > > > Thanks > > Sau! > > > > > > > > Forwarded Message > > Subject: CVE References in LPs are messed up after centos feature > > branch rebase > > Date: Fri, 13 Dec 2019 00:30:26 + > > From: Khalil, Ghada > > To: Saul Wold > > > > > > > > Hi Saul, > > > > The CVE References in about 15 LPs are now messed up after the rebase of > > the f-centos8 feature branch. The rebase updated a large # of launchpads > > and somehow automatically added CVE references (from a subset of bugs) > > to all of them. Any idea what is going on here? > > > > Here are some examples: > > > > https://bugs.launchpad.net/starlingx/+bug/1844579 > > > > Originally had no CVE References. Now it has 3 references. > > > > https://bugs.launchpad.net/starlingx/+bug/1849200 > > > > Originally only had CVE-2018-15686 as a CVE Reference. Now it has all > > the recently fixed CVEs linked to this bug. > > > > Snapshot from the full activity log: > > > > Here is the query that shows that all the bugs that were picked up in > > the rebase now have CVE links: > > > >
Re: [OpenStack-Infra] Fwd: CVE References in LPs are messed up after centos feature branch rebase
On Fri, Dec 13, 2019, at 8:48 AM, Saul Wold wrote: > > Hello Infra team: > > Apparently something got messed up with Launchpad and updating a number > of starlingx repos with a feature branch. > > I was following the methodology of updating a feature branch with > changes from master via merges and I guess when I pushed that to gerrit > and it merged, it caused some Launchpad ugliness. See email below. > > Thoughts? I think what happened here is you merged bug fixes (in this case cve bug fixes) from master into a feature branch. Then when you pushed that merge commit and merged it, the bot noticed that those bug fixes had merged to the feature branch and commented with those details on the bug. I believe this is "correct" behavior from the bot. Is the issue the existence of comments like https://bugs.launchpad.net/starlingx/+bug/1844579/comments/18 on the bugs? Or is there some other metadata that is being added that I am missing? If we don't want comments like that to appear you'd need to modify your merged trees so that bug fixes don't go from master into the feature branch. Or we'd need to come up with some rule set we can apply to the bot to filter bugs out in certain circumstances. > > Thanks > Sau! > > > > Forwarded Message > Subject: CVE References in LPs are messed up after centos feature > branch rebase > Date: Fri, 13 Dec 2019 00:30:26 + > From: Khalil, Ghada > To: Saul Wold > > > > Hi Saul, > > The CVE References in about 15 LPs are now messed up after the rebase of > the f-centos8 feature branch. The rebase updated a large # of launchpads > and somehow automatically added CVE references (from a subset of bugs) > to all of them. Any idea what is going on here? > > Here are some examples: > > https://bugs.launchpad.net/starlingx/+bug/1844579 > > Originally had no CVE References. Now it has 3 references. > > https://bugs.launchpad.net/starlingx/+bug/1849200 > > Originally only had CVE-2018-15686 as a CVE Reference. Now it has all > the recently fixed CVEs linked to this bug. > > Snapshot from the full activity log: > > Here is the query that shows that all the bugs that were picked up in > the rebase now have CVE links: > > https://bugs.launchpad.net/starlingx/+bugs?field.searchtext==-importance%3Alist=NEW%3Alist=OPINION%3Alist=INVALID%3Alist=WONTFIX%3Alist=EXPIRED%3Alist=CONFIRMED%3Alist=TRIAGED%3Alist=INPROGRESS%3Alist=FIXCOMMITTED%3Alist=FIXRELEASED%3Alist=INCOMPLETE_WITH_RESPONSE%3Alist=INCOMPLETE_WITHOUT_RESPONSE_option=any=_reporter=_commenter==_subscriber==in-f-centos8_combinator=ANY_cve.used=_cve=on_dupes.used=_me.used=_patch.used=_branches.used=_branches=on_no_branches.used=_no_branches=on_blueprints.used=_blueprints=on_no_blueprints.used=_no_blueprints=on=Search > > *Ghada Khalil*, Manager, Titanium Cloud, *Wind River* > direct 613.270.2273 skype ghada.khalil.ottawa > > 350 Terry Fox Drive, Suite 200, Kanata, ON K2K 2W5 > > > ___ > OpenStack-Infra mailing list > OpenStack-Infra@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Resigning from infra-core
On Mon, Dec 9, 2019, at 3:28 PM, Joshua Hesketh wrote: > Hello all, > > I've been dreading writing this for a while now but all good things > must come to an end. Due to changes at work and my inability to sustain > any significant contributions to the project I think that regrettably > it is time to resign from infra-core. > > It has been a wonderful experience contributing to OpenStack and > OpenDev I'll miss working with everybody and it has been an absolute > pleasure to get to know some of you both professionally and personally. > I will certainly miss you all and cannot understate how thrilled I am > to have had this chance to work with everyone. It has been a highlight > of both my career and in some cases my life. > > Having said that, I will still be around, so please do not hesitate to > contact me should you need my input on anything or if there is > something I can do to help. I also intend to still be involved with > Zuul in whatever capacity I can. > > I hope that our paths will cross again in the near future, and wish you > all all the best with the projects going forward. > > Kind regards, > Josh Thank you for all the help over the years. I too have enjoyed working with you and am sad to make this change official. If you ever make it back towards openstack/opendev we'd be happy to add you into the core group again. Also thank you for giving us the best Gerrit username ever: Turbo Hipster. I wish you luck with your new endeavors and hope to see you around, Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for December 10, 2019
We will have our weekly team meeting at 19:00UTC December 10, 2019 in #openstack-meeting. We will use this agenda: == Agenda for next meeting == * Announcements ** No meeting December 24, 2019 and December 31, 2019 * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Possible gitea/go-git bug in current version of gitea we are running https://storyboard.openstack.org/#!/story/2006849 *** Bug seems to remain present after Gitea 1.10 upgrade * General topics ** Trusty Upgrade Progress (clarkb 20191210) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20191210) *** Infra-root needs to create AFS volumes. ** dib/nodepool container (ianw 20191210) *** http://lists.openstack.org/pipermail/openstack-infra/2019-November/006529.html ** Discussion on retiring services that are largely unmaintained or unused (clarkb 20191210) *** Want to start this discussion as services like Ask are barely on life support and we should probably clean them up. *** https://etherpad.openstack.org/infra-service-list * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] OpenDev Independence and Governance
On Tue, Dec 3, 2019, at 12:05 PM, Clark Boylan wrote: > Hello everyone, > > The OpenDev project has been running semi formally for about a year > now. During this time we have tried to accomodate the > needs of our various constituent projects, but we've still (for the > most part) formally operated under OpenStack's governance. > In order to better serve the projects we host that are not OpenStack we > think it is important for OpenDev to become an > independent entity with its own governance structure. > > In James Blair's winterscale email [0] he suggested that we create a > governing council made up of the OpenDev PTL and > a representative from each of the OpenStack Foundation's official > projects that currently consume OpenDev resources > (currently OpenStack itself, Airship, and Zuul). This suggestion > creates two levels of governance for the OpenDev team. > > The first is the position of PTL for the OpenDev project. If we want to > continue to manage this position as we've managed it > for the OpenStack Infra team, then we can have elections for the > position every 6 months. The nominee pool and electorate > would be individuals that have contributed changes to OpenDev in the > last 12 months. > > For the council, membership would be small, but I think demands on this > group would also be minimal. Ideally the OpenDev team > would be left to figure out technical details for services and this > council would be used as a check on service changes or > other behavioral updates that affect how OpenDev's users interact with > the system. Since this group would be starting with > an even number of individuals we may need to determine tie breaker > requirements upfront. Also, we may want to consider > if the "else" group of OpenDev users need a voice. Individuals > representing constituent projects should be nominated by > their project leadership. > > As for next steps, I think we want to sort out these governance details > to where we are generally happy with them, then we > can make the formal request to the OpenStack TC to pull anchor and sail > a bit more independently. > > Feedback is more than welcome, > Clark > > [0] http://lists.openstack.org/pipermail/openstack-dev/2018-May/130896.html Incorporating feedback I've produced this revised proposal: The OpenDev project will be governed by two entities. The first is the service coordinator. Responsibilities for the service coordinator are essentially the same of the existing Infra team PTL. They coordinate work of contributors and act as a tie breaker when clear consensus isn't found. The service coordinator is elected every 6 months. The nominee pool and electorate are individuals that have contributed changes to OpenDev in the last 12 months. The second is an advisory board made up of representatives from OpenDev's user base of projects and organizations that contribute compute resources. This advisory board provides a formal location for both our users and contributing orgs to express their needs to the OpenDev project. This creates a clear contact point for feedback on priorities and direction. Their input will help ensure that the OpenDev project is a good steward of the resources provided to it and that user needs are being addressed. Contributing orgs and user projects are not required to join the advisory board, but are encouraged to do so. Individuals on the board would be selected by their own constituency as that constituency feels is appropriate. The advisory board will also serve as a point of contact for the OpenDev project when making changes that may be disruptive. The intent is to create bidirectional communication between OpenDev and the advisory board. How does this look? Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] OpenDev Independence and Governance
Hello everyone, The OpenDev project has been running semi formally for about a year now. During this time we have tried to accomodate the needs of our various constituent projects, but we've still (for the most part) formally operated under OpenStack's governance. In order to better serve the projects we host that are not OpenStack we think it is important for OpenDev to become an independent entity with its own governance structure. In James Blair's winterscale email [0] he suggested that we create a governing council made up of the OpenDev PTL and a representative from each of the OpenStack Foundation's official projects that currently consume OpenDev resources (currently OpenStack itself, Airship, and Zuul). This suggestion creates two levels of governance for the OpenDev team. The first is the position of PTL for the OpenDev project. If we want to continue to manage this position as we've managed it for the OpenStack Infra team, then we can have elections for the position every 6 months. The nominee pool and electorate would be individuals that have contributed changes to OpenDev in the last 12 months. For the council, membership would be small, but I think demands on this group would also be minimal. Ideally the OpenDev team would be left to figure out technical details for services and this council would be used as a check on service changes or other behavioral updates that affect how OpenDev's users interact with the system. Since this group would be starting with an even number of individuals we may need to determine tie breaker requirements upfront. Also, we may want to consider if the "else" group of OpenDev users need a voice. Individuals representing constituent projects should be nominated by their project leadership. As for next steps, I think we want to sort out these governance details to where we are generally happy with them, then we can make the formal request to the OpenStack TC to pull anchor and sail a bit more independently. Feedback is more than welcome, Clark [0] http://lists.openstack.org/pipermail/openstack-dev/2018-May/130896.html ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Openstack console not working
On Wed, Nov 27, 2019, at 10:47 PM, Ali Hosseinbeigi wrote: > Hi, > > Ive installed openstack (train) via MAAS and juju (Conjure-up, nova-kvm ). > > Every thing works fine but I have problem with instance console. > > When I click on console the message “Instance Console Unable to load > console. Please reload page to try again.” appears. > > Is there anyone to helm me? Sent this to the first thread too, but to ensure it gets seen I'll send it to this one as well. Hope this helps. This mailing list is used to coordinate the efforts to produce OpenStack software: things like Gerrit and Zuul. Unfortunately this means we are not in a great position to help debug OpenStack installations. For that you should send mail to openstack-disc...@lists.openstack.org instead. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Infra Team Meeting Agenda for December 3, 2019
We will meet at 19:00UTC on December 3 (today) in #openstack-meeting with this agenda. Sorry for the late send. I was busy traveling yesterday. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Possible gitea/go-git bug in current version of gitea we are running https://storyboard.openstack.org/#!/story/2006849 * General topics ** Trusty Upgrade Progress (clarkb 20191203) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20191203) *** Infra-root needs to create AFS volumes. ** dib/nodepool container (ianw 20191203) *** http://lists.openstack.org/pipermail/openstack-infra/2019-November/006529.html ** Discussion on retiring services that are largely unmaintained or unused (clarkb 20191203) *** Want to start this discussion as services like Ask are barely on life support and we should probably clean them up. *** https://etherpad.openstack.org/infra-service-list * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Problem with Openstack
On Wed, Nov 27, 2019, at 5:42 AM, Ali Hossein Beigi wrote: > > Hi, > > Ive installed openstack (train) via MAAS and juju (Conjure-up, nova-kvm ). > > Every thing works fine but I have problem with instance console. > > When I click on console the message “Instance Console Unable to load > console. Please reload page to try again.” appears. > > Is there anyone to helm me? This mailing list is used to coordinate the efforts to produce OpenStack software: things like Gerrit and Zuul. Unfortunately this means we are not in a great position to help debug OpenStack installations. For that you should send mail to openstack-disc...@lists.openstack.org instead. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for November 26, 2019
We will meet in #openstack-meeting at 19:00UTC November 26 with this agenda: == Agenda for next meeting == * Announcements ** Holiday week for those in the USA * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Possible gitea/go-git bug in current version of gitea we are running https://storyboard.openstack.org/#!/story/2006849 * General topics ** Trusty Upgrade Progress (clarkb 20191126) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20191126) *** Infra-root needs to create AFS volumes. ** Installing tox with py3 in base images runs some envs with py3 now - update (ianw 20191125) *** http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010957.html ** Submariner on opendev.org (dgroisma, mkolesni) ** dib/nodepool container (ianw 20191125) *** ianw to do writeup before ** Discussion on retiring services that are largely unmaintained or unused (clarkb 20191126) *** Want to start this discussion as services like Ask are barely on life support and we should probably clean them up. *** https://etherpad.openstack.org/infra-service-list * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] helping openstack-infra
On Mon, Nov 25, 2019, at 4:15 AM, Sorin Sbarnea wrote: > Hi! > > In response to > https://governance.openstack.org/tc/reference/upstream-investment-opportunities/2019/community-infrastructure-sysadmins.html > would like to state that I am more than willing to help the openstack-infra > team with sysadmin tasks needed. > > I am fully aware this will require me investing more time on infra > specific tasks but I am glad it do have the support from my > organization for doing this. > > As I was really happy about my last year experience with others from > infra and I valued a lot their help on various tasks, I want to also > return the favor and help with other chores or features we may need do > to make our CI even better. > > So mainly, just let me know what needs should be done? Current priority efforts include OpenDev-ification of services and updating config management for puppet managed services to ansible and containers. To opendevify services we are updating them to be hosted at opendev.org domains as well as updating theming if necessary to incorporate the opendev logo and color scheme. In many cases we want to install redirects from the old openstack.org names to the new opendev.org name. For services with SSL/TLS this is likely the trickiest bit of the conversion as our plan is to use LetsEncrypt for opendev.org names. Thankfully, we've sorted out a plan [0] for managing openstack.org names with LetsEncrypt certs as well. For the config management updates we've been converting deployments of services from puppet to ansible and in many cases having ansible drive docker-compose to do the actual deployments. This process typically starts by determining if we can use an upstream image or not. If not we add a Dockerfile and corresponding image build jobs to the opendev/system-config repo. Then we can add jobs to test deployment of those containers and finally deploy to production via this method. The great thing about this process is we can fully test the deployment easily and there are many examples of that in system-config now (see Gitea). Whether or not it makes more sense to convert a service to opendev or update its config management first, likely depends on how difficult it is to set up SSL/TLS for multiple domains. My hunch is that for most services doing the config management update with the SSL/TLS needs in mind is likely easiest. If you'd like a concrete place to start I would suggest taking a service like ethercalc or etherpad, udpate its config management to ansible + docker-compose, implement test jobs as others have, then when all that is happy you can work with an infra root to replace our deployment of these services in production. Then if we haven't already done it in conjunction with the config management updates we can update the SSL/TLS setup and convert to an opendev service. Another different but slightly related OpenDev topic is to start implementing tooling so that top level orgs can manage their repositories directly. At the PTG we discussed doing this via a meta project per repo that determines who is allowed to make those changes, then via ansible of some sort implement those updates when the appropriate approvers have ACKed the change. This likely needs a bit of design work just to be sure we are all in agreement of the plan. A lightweight spec would be helpful. [0] https://opendev.org/opendev/infra-specs/src/branch/master/specs/retire-static.rst#opendev-infrastructure-migration Hope this helps. Feel free to dig in or ask questions and I'll do my best to expand on this topic. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] [zuul-jobs] configure-mirrors: deprecate mirroring configuration for easy_install
On Mon, Nov 25, 2019, at 5:38 AM, Paul Belanger wrote: > On Mon, Nov 25, 2019 at 04:02:13PM +1100, Ian Wienand wrote: > > Hello, > > > > Today I force-merged [5] to avoid widespread gate breakage. Because > > the change is in zuul-jobs, we have a policy of annoucing > > deprecations. I've written the following but not sent it to > > zuul-announce (per policy) yet, as I'm not 100% confident in the > > explanation. > > > > I'd appreciate it if, once proof-read, someone could send it out > > (modified or otherwise). > > > > Thanks, > > > Greetings! > > Rather then force merge, and potential break other zuul installs. What > about a new feature flag, that was still enabled but have openstack base > jobs disabled? This would still allow older versions of setuptools to > work I would guess? > I think the ship has sailed and the change has already been force merged. That said setuptools isn't something that you can easily control versioning of. For example when you create a virtualenv the version of setuptools in the virtualenv is automatically updated to the latest version. For this reason I think the merge was the best course of action. > That said, ansible Zuul is not affected as we currently fork > configure-mirrors for our open puproses, I'll check now that we are also > not affected. > ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for November 19, 2019
Hello, We will have our weekly team meeting tomorrow, November 19 at 19:00UTC in #openstack-meeting. Note that I may be a little late to the meeting myself as I have poorly scheduled a dentist visit with the recent DST change. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** LetsEncrypt issued cert for opendev.org https://review.opendev.org/#/c/694181/12 *** Possible gitea/go-git bug in current version of gitea we are running https://storyboard.openstack.org/#!/story/2006849 * General topics ** Trusty Upgrade Progress (clarkb 20191119) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20191119) *** Ajaeger started writing some changes https://review.opendev.org/#/q/topic:static-services+(status:open+OR+status:merged) *** Infra-root needs to create AFS volumes. ** ask.openstack.org apache2 crashing daily during logrotate (clarkb 20191119) *** Needs a volunteer to further debug and find a permanent fix *** Some logs: http://paste.openstack.org/show/785843/ *** Any progress on this? ** Installing tox with py3 in base images runs some envs with py3 now (frickler 20191115) *** See http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-11-15.log.html#t2019-11-15T11:10:57 ff. * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Infra Shanghai PTG Recap
Going to do my best to summarize the events of the PTG for the Infra team. Notes were taken at the event in this etherpad: https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019. Feel free to ask followup questions and I will do my best to answer them. Our PTG started with Gitea as two local Gitea maintainers were able to make it to the event for half a day to talk to us. They are aware of the performance problems we have experienced and it is important to them to get these issues fixed as we are not the only users that have noticed. There is also work in progress to implement elasticsearch hosted indexes for code search and issue searching. The person working on this for code search is no longer working on this PR so monty and jim will see if they can revive it. When asked where the Gitea project could use help it seemed these two items were top of the list. If you want to get involved they use discord for communictions: https://discordapp.com/channels/322538954119184384/322538954119184384. Other things to keep in mind with Gitea is they are pushing to be self hosted. There are web hooks that could be used to build a Gitea driver for Zuul. This may be useful if we decide to perform third party CI for Gitea when they are self hosted (and otherwise if people want to use Zuul and Gitea together). Gitea exposes Prometheus metrics which we may want to start recording in order to measure performance improvements over time as upstream merges fixes. I was unable to find a Prometheus to statsd converter though so we may need some infrastructure for this. Next up was OpenDev discussions. We talked about decoupling Zuul driven deployments from our desire to implement automation of project management for our hosted orgs. This was openstack and zuul et al could potentially start self managing project creation prior to Zuul driven CD. To do this we would likely have a cron run an ansible playbook. Then when Zuul driven CD is ready on our side have it run that playbook instead of cron. We also talked about sending out the governance email (this is fairly high on my todo list). Related to governance was formalizing the relationship a bit more with our infrastructure donors and having proper liaisons (we have informal liaisons today). Then using these liaisons to help push the benefits of having us consume their clouds (we find bugs before customers, ensure that cloud can handle fairly high level of load, etc). In our containerization efforts we heard from Monty that podman should be ready to go. There is minor concern that the Ubuntu PPA is pulling development updates rather than stable releases, but with newer software like podman this may be desirable. We'll avoid rolling it out everywhere until we have proper support for speculative testing and gating of containers with podman in our Zuul jobs. Until then we'll keep it to the review(-dev) images as they don't change often. Recently we've had trouble where our nodepool builder hosts are unable to bootstrap new images because the host platform doesn't have zypper (bionic) or can't uncompress fedora 31 RPM packages (xenial). It was pointed out that something like 50% of possible DIB + build platform choices result in being unable to build one image or another. To help address this Monty and Jim had the idea that we could bypass direct bootstrapping and simply unpack upstream docker images. Then we'd have a filesystem we can chroot into that includes all necessary tools for that platform. To do this we do need to figure out kernel management (as well as GRUB?) because container images don't typically include this data. For the static.openstack.org rebuild mnaser has volunteered to help with job changes on the OpenStack side of things. Ajaeger has also gotten this process started. That means we need to go ahead and start provisioning AFS volumes to publish into. For the DNS updates we talked about using the static.o.o hosted names as a proving ground for the larger plan to CNAME all openstack.org names into opendev.org. We don't need to batch that all up, but can if we are happy with the results with static's previously hosted content. We also talked about translation services as Zanata is no longer supported. The i18n team is interested in using weblate, we (probably me?) need to reach out to the weblate team to see if hosting an open source project the size of openstack is outside of their scope for their normal open source hosting. If it is then we can work with the OSF to see if using the paid hosting platform for weblate is viable. If not we also have the option of possibly hosting our own weblate. Finally there were discussions about Zuul specific topics. This email is already quite long so I'll keep it scoped to the OpenDev/Infra topics. If there is interest in a similar email re Zuul topics I can probably send one out to the Zuul list. Clark ___
[OpenStack-Infra] Meeting Agenda for November 12, 2019
We will meet in #openstack-meeting at 19:00 UTC on November 12, 2019 with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev * General topics ** PTG and Summit Recap (clarkb 20191112) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 ** Trusty Upgrade Progress (clarkb 20191029) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20191112) *** At PTG mnaser volunteered to work with us on the openstack side. *** We'll start DNS changes with static's hosted names prior to doing any mass migration. ** ask.openstack.org apache2 crashing daily during logrotate (frickler 2019) *** Needs a volunteer to further debug and find a permanent fix *** Some logs: http://paste.openstack.org/show/785843/ * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for October 29, 2019
We will meet in #openstack-meeting at 19:00UTC on October 29, 2019 with this agenda: == Agenda for next meeting == * Announcements ** Summit and PTG next week. Meeting on November 5 Cancelled. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev * General topics ** Graphite update (clarkb 20191029) *** Retention for stats.timers reduced. Files updated and rechowned. ** PTG Planning (clarkb 20191029) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 *** https://www.openstack.org/ptg/#tab_schedule PTG Schedule ** Cleaning up OpenSuse 150 images (clarkb 20191029) *** The 150 images in nodepool were replaced by the 15 images. Need to update jobs so that we can remove the 150 images. *** https://review.opendev.org/#/q/topic:use-opensuse-15 ** Trusty Upgrade Progress (clarkb 20191029) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20191029) *** Review spec: https://review.opendev.org/683852 *** Sign up for tasks at https://etherpad.openstack.org/p/static-services *** https://etherpad.openstack.org/p/openstack-org-dns * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Team Meeting Agenda for October 22, 2019
We will be meeting on October 22, 2019 at 19:00UTC in #openstack-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev * General topics ** PTG Planning (clarkb 20191022) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 *** https://www.openstack.org/ptg/#tab_schedule PTG Schedule ** Trusty Upgrade Progress (clarkb 20191022) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20191022) *** Review spec: https://review.opendev.org/683852 *** Sign up for tasks at https://etherpad.openstack.org/p/static-services *** https://etherpad.openstack.org/p/openstack-org-dns * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Infra Team Meeting Agenda for October 15, 2019
We will meet at 19:00UTC on October 15, 2019 (tomorrow) in #openstack-meeting with this agenda: == Agenda for next meeting == * Announcements ** OpenStack Release this Week (20191015) * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev * General topics ** Trusty Upgrade Progress (clarkb 20191015) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20191015) *** Review spec: https://review.opendev.org/683852 *** Sign up for tasks at https://etherpad.openstack.org/p/static-services *** https://etherpad.openstack.org/p/openstack-org-dns Brought this up with the OSF and they had no initial concerns though jbryce said he wouldn't get to review it until this week. ** CentOS 8 (clarkb 20191015) *** Do we need to consider pushing projects to switch labels when 8 is available? *** zbr is updating the configure-mirror role to support CentOS 8 *** We believe that the NetworkManager issue has been tracked down and fixed (thank you ianw) The issue that prevents ipv4 from being configured is related to glean up'ing the interface between NetworkManager starts. We stopped glean from doing this. The issue that prevented ipv6 from being configured was related to our RA solicit delay value. Which we've removed. ** PTG Planning (clarkb 20191015) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 *** https://www.openstack.org/ptg/ schedule can be found here under the schedule tab. * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Agenda for Team Meeting on October 8, 2019
We will meet on October 8 at 19:00UTC in #openstack-meeting with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev * General topics ** Trusty Upgrade Progress (clarkb 20191008) *** Wiki updates ** static.openstack.org (ianw,corvus,mnaser,fungi 20191008) *** Review spec: https://review.opendev.org/683852 *** Sign up for tasks at https://etherpad.openstack.org/p/static-services *** https://etherpad.openstack.org/p/openstack-org-dns ** CentOS 8 (clarkb 20191008) *** Do we need to consider pushing projects to switch labels when 8 is available? *** We continue to have glean + NetworkManager + Red Hat distro problems on ipv6 only clouds. https://review.opendev.org/#/c/686474/3 https://review.opendev.org/#/c/686749/5 ** PTG Planning (clarkb 20191008) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 *** http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009733.html Strawman schedule * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Infra Meeting Agenda for October 1, 2019
We will be meeting at 19:00UTC on October 1, 2019 with this agenda: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev * General topics ** Trusty Upgrade Progress (clarkb 20191001) *** Wiki updates ** static.openstack.org (ianw 20191001) *** Review spec: https://review.opendev.org/683852 *** Sign up for tasks at https://etherpad.openstack.org/p/static-services ** PTG Planning (clarkb 20191001) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 *** http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009733.html Strawman schedule * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Infra Meeting Cancelled on September 24, 2019
Per our last meeting [0] we are cancelling tomorrow's meeting. Many of us are traveling and will be unable to attend the normal meeting time. [0] http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-09-17-19.01.log.html#l-19 See you next week! Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for September 17, 2019
Hello, We will have our weekly meeting on September 17 at 19:00UTC with this agenda: == Agenda for next meeting == * Announcements ** Clarkb at Ansiblefest next week. Likely need volunteer meeting chair. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev * General topics ** Trusty Upgrade Progress (clarkb 201900917) *** Wiki updates ** static.openstack.org (ianw 20190917) *** Sign up for tasks at https://etherpad.openstack.org/p/static-services ** PTG Planning (clarkb 20190917) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for September 10, 2019
We will meet on September 10, 2019 at 19:00UTC with this agenda: == Agenda for next meeting == * Announcements ** clarkb PTL for Ussuri cycle. Planning to not run next cycle. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev * General topics ** Trusty Upgrade Progress (clarkb 201900910) *** Wiki updates ** static.openstack.org (ianw 20190910) *** Sign up for tasks at https://etherpad.openstack.org/p/static-services ** AFS mirroring status (ianw 20190910) *** afs02.dfw.openstack.org was down and rebooted. ** Project Renaming September 16 (clarkb 20190910) ** PTG Planning (clarkb 20190910) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 ** Volume of files from ARA html reports is problematic (dmsimard 20190910) *** https://etherpad.openstack.org/p/Vz5IzxlWFz *** We disabled root ARA report in jobs *** Added better sharding to upload-logs-swift to spread out the objects better * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Report from Gerrit User Summit
On Wed, Sep 4, 2019, at 9:53 AM, James E. Blair wrote:
> Hi,
>
> Monty and I attended the Gerrit User Summit and hackathon last week. It
> was very productive: we learned some good information about upgrading
> Gerrit, received offers of help doing so if we need it, formed closer
> ties with the Gerrit community, and fielded a lot of interest in Reno
> and Zuul. In general, people were happy that we attended as
> representatives of the OpenDev/OpenStack/Zuul communities and (re-)
> engaged with the Gerrit community.
>
> Gerrit Upgrade
> --
>
> We learned some practical things about upgrading to 3.0:
>
> * We can turn off rebuilding the secondary index ("reindexing") on
> startup to speed both or normal restarts as well as prevent unwanted
> reindexes during upgrades. (Monty pushed a change for this.)
>
> * We can upgrade from 2.13 -> 2.14 -> 2.15 -> 2.16 during a relatively
> quick downtime. We could actually do some of that while up, but Monty
> and I advocate just taking a downtime to keep things simple.
>
> * We should, under no circumstances, enable NoteDB before 2.16. The
> migration implementation in 2.15 is flawed and will cause delays or
> errors in later upgrades.
>
> * Once on 2.16, we should enable NoteDB and perform the migration. This
> can happen online in the background.
>
> * We should GC the repos before starting, to make reindexing faster.
>
> * We should ensure that we have a sufficiently sized diff cache, as that
> Gerrit will be able to re-use previously computed patchset diffs when
> reindexing. This can considerably speed an onlide reindex.
>
> * We should probably run 2.16 in production for some time (1 month?) to
> allow users to acclimate to polygerrit, and deal with hideCI.
>
> * Regarding hideCI -- will someone implement that for polygerrit? will
> it be obviated by improvements in Zuul reporting (tagged or robot
> comments)? even if we improve Zuul, will third-party CI's upgrade?
> do we just ignore it?
>
> * The data in the AccountPatchReviewDb are not very important, and we
> don't need to be too concerned if we lose them during the upgrade.
>
> * We need to pay attention to H2 tuning parameters, because many of the
> caches use H2.
>
> * Luca has offered to provide any help if we need it.
>
> I'm sure there's more, but that's a pretty good start. Monty has
> submitted several changes to our configuration of Gerrit with the topic
> "gus2019" based on some of this info.
This is excellent information and makes the Gerrit upgrade seem far more
doable. Thank you for this.
>
> Gerrit Community
>
Snip
> Reno
>
Snip
> Zuul
>
>
> Zuul is happily used at Volvo by the propulsion team at Volvo (currently
> v2, working on moving to v3) [2]. Other teams are looking into it.
>
> The Gerrit maintainers are interested in using Zuul to run Gerrit's
> upstream CI system. Monty and I plan on helping to implement that.
>
> We spoke at length with Edwin and Alice who are largely driving the
> development of the new "checks" API in Gerrit. It is partially
> implemented now and operational in upstream Gerrit. As written, we
> would have some difficulty using it effectively with Zuul. However,
> with Zuul as a use case, some further changes can be made so that I
> think it will integrate quite well, and with more work could be a quite
> nice integration.
>
> At a very high level, a "checker" in Gerrit represents a single
> pass/fail result from a CI system or code analyzer, and must be
> configured on the project in advance by an administrator. Since we want
> Zuul to report the result of each job it runs on a change, and we don't
> know that set of jobs until we start, the current implementation doesn't
> fit the model very well. For the moment, we can use the checks API to
> report the overall buildset result, but not the jobs. We can, of
> course, still use Gerrit change messages to report the results of
> individual jobs just as we do now. But ideally, we'd like to take full
> advantage of the structured data and reporting the checks API provides.
>
> To that end, I've offered to write a design document describing an
> implementation of support for "sub-checks" -- an idea which appeared in
> the original checks API design as a potential follow-up.
>
> Sub-checks would simply be structured data about individual jobs which
> are reported along with the overall check result. With this in place,
> Zuul could get out of the business of leaving comments with links to
> logs, as each sub-check would support its own pass/fail, duration, and
> log url.
>
> Later, we could extend this to support reporting artifact locations as
> well, so that within Gerrit, we would see links to the log URL and docs
> preview sites, etc.
>
> There is an opportunity to do some cross-repo testing between Zuul and
> Gerrit as we work on this.
>
> Upstream Gerrit's Gerrit does not have the SSH event stream
[OpenStack-Infra] Infra Team Meeting September 3, 2019 at 19:00 UTC
Sorry for getting this out late. We will meet with this agenda: == Agenda for next meeting == * Announcements ** OpenStack election season is upon us * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev * General topics ** Trusty Upgrade Progress (clarkb 201900903) *** Wiki updates ** static.openstack.org (ianw 20190903) *** Sign up for tasks at https://etherpad.openstack.org/p/static-services ** AFS mirroring status (ianw 20190903) *** Did debugging additions help? *** Do we think rsync updates are to blame? Perhaps in newer rsync on Bionic? ** Project Renaming September 16 (clarkb 20190903) ** PTG Planning (clarkb 20190903) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] [infra][nova] Corrupt nova-specs repo
On Mon, Sep 2, 2019, at 1:45 AM, Chance Huss wrote: > ___ > OpenStack-Infra mailing list > OpenStack-Infra@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra Can you provide more information? I am able to clone nova-specs from all eight of our gitea backends as well as github and they all fsck cleanly. The gitea web ui also seems to render the base repo page for nova-specs on all eight backends too. Where is the corrupt repo? How are you detecting the corruption or how is it reported? Any info like that will be useful to debug this for you. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] is basic auth broken on our gerrit: review.opendev.org?
On Mon, Sep 2, 2019, at 9:09 AM, Jeremy Stanley wrote: > On 2019-09-02 16:58:48 +0100 (+0100), Sorin Sbarnea wrote: > [...] > > Considering that from 2.14 only BasicAuth will be supported, it > > makes sense to make the move now. > > Yes, I find this a compelling argument, if only to flush out > problems before we upgrade to 2.14. No objections to changing it; however, if this is expected to be a breaking change (basic auth means no more digest auth) we'll likely need to communicate that similarly to a gerrit upgrade. > > > PS. I stopped using pygerrit2 and I am using requests directly, as > > I did not find many benefits in pygerrit so far. > > Pretty much all the OpenDev tools also just interface directly with > the Gerrit API. Same for projects like Gertty. > -- > Jeremy Stanley ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] is basic auth broken on our gerrit: review.opendev.org?
On Mon, Sep 2, 2019, at 8:58 AM, Sorin Sbarnea wrote: > it would be really nice if we could enable HTTPBasicAuth because at > this moment talking with multiple gerrit servers seems to be a real > challenge. > > For example: > * review.rdoprpject.org server works only with BasicAuth and also > requires custom url suffix "/r". > * review.opendev.org works only with Digest auth This doesn't seem to be that difficult: http://paste.openstack.org/show/769640/ Note the custon /r suffix is a choice that was made by the hosts of that Gerrit iirc and not a default Gerrit behavior. Basically they moved the root of the Gerrit servers a path level down under /r. > > Considering that from 2.14 only BasicAuth will be supported, it makes > sense to make the move now. ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Questions: openstack-ansible-ops -> multi-node-aio
On Thu, Aug 29, 2019, at 10:12 AM, Ich bins wrote: > After upgrading "pip" to newest version, now the following error occurs: > > ERROR: Package 'more-itertools' requires a different Python: 2.7.15 not > in '>=3.4' > > > > > Am Do., 29. Aug. 2019 um 19:08 Uhr schrieb Ich bins > : > > Hi folks, > > > > I have some short questions about the multi node AIO deployment. > > > > - Is it possible to deploy the virtual environment on just one disk ? > > I have a large disk, which is pre-formatted with one big LVM called > > "/dev/mapper/cloudbox001-vg/root" and no unformatted space on it. There is > > enough space (aprox. 2 TB) for deployment, but it fails to deploy. > > > > - Which environment variables have to be configured ? > > If I start build.sh without any variables, it ends up in > > > > https://pastebin.com/pyFR7Mzi > > > > Regards and take care. > > > > Will The OpenStack Infra team is predominantly tasked with building and operating tools to build the openstack software. Unfortunately, this means we don't have a ton of experience with tools like openstack ansible. You probably want to send your queries to openstack-disc...@lists.openstack.org instead where you should have a mix of developers, operators, and users of openstack including those that build and deploy OSA. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for August 27, 2019
We will meet at 19:00 UTC on August 27, 2019 to discuss this agenda: == Agenda for next meeting == * Announcements ** OpenStack election season is upon us * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev * General topics ** Trusty Upgrade Progress (clarkb 201900827) *** Wiki updates ** static.openstack.org (ianw 20190827) *** service audit -> https://etherpad.openstack.org/p/static-services *** future does not seem to be uploading too/serving from a static host *** move redirects to dockered haproxy instance https://review.opendev.org/#/c/677903/5 -> split haproxy out to be more generic from gitea https://review.opendev.org/678159 -> Add a service load balancer, add existing redirects, test *** longer term goal move others to afs publishing? *** convert to ansible/bionic host post logs.o.o expiration? ** AFS mirroring status (ianw 20190827) *** Debian buster updates not populated by reprepro but are assumed to be present by our mirror setup roles. *** fedora issues again (http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-08-26.log.html#t2019-08-26T07:25:16) -- pattern? ** PTG Planning (clarkb 20190827) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 ** Next rename window (Kayobe rename(s) needed for Train release inclusion) * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Failed to SSH login into the gerrit(Too many concurrent connections )
On Fri, Aug 23, 2019, at 9:40 AM, Rikimaru Honjo wrote: > Hello, > > I have my zuulv3 instance for 3rd party CI. > > I stopped & restarted it several times to test my configuration > yesterday. > As aresult, my zuulv3 instance started to fail to connect the > gerrit(review.opendev.org). > The following error was occurred at that time. > > > Received disconnect from 104.130.246.32 port 29418:12: Too many concurrent > > connections (64) - max. allowed: 64 > > I tried to run ssh command[1] manually. It was also failed by same reason. > And, I also tried it on other machine. The result was the same. > > How can I resolve this issue? There is only one user with 64 connections currently so I am going to assume this is your third party CI user (it is NTT SystemFault MasakariIntegration CI). I've gone ahead and closed all 64 existing connections in Gerrit. This is typically caused by ssh clients that don't properly close their ssh connection with gerrit. At one point this happened because of a bug in Zuul. That bug has long since been fixed; you should ensure that you are running an up to date version of zuul and paramiko to avoid this. Firewalls that timeout connections may also cause this to happen. Also for future debugging it helps to get the client name and/or ID so that we can be sure we are identifying the correct accounts and connections on the Gerrit side of things. > > [1] > e.g. "gerrit stream-events", "git-upload-pack " > > Best regards, > -- > _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/ > Rikimaru Honjo > E-mail:honjo.rikim...@ntt-tx.co.jp ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for August 20, 2019
We will have our weekly team meeting at 19:00UTC tomorrow August 20, 2019. This is the agenda we will follow: == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev * General topics ** Trusty Upgrade Progress (clarkb 201900820) *** Job logs are now in swift ** AFS mirroring status (ianw 20190820) *** Debian buster updates not populated by reprepro but are assumed to be present by our mirror setup roles. *** Fedora seems to have stopped updated again ** PTG Planning (clarkb 20190820) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 ** New backup server (ianw 20190820) *** https://review.opendev.org/#/c/675537 ** Intermittent Lack of IPv4 in Limestone (clarkb 20190820) * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] opensource infra: server sizes
On Mon, Aug 19, 2019, at 9:36 AM, Shadi Akiki wrote: > > the sizing details for control plane servers are not really listed anywhere > > If they're not listed anywhere, I suppose that nobody follows up on the > sizing until something breaks? > > > Sometimes the cloud provider will provide us with custom flavors, or ask us > > to use a particular variant > > Why would they ask for a particular variant? Is it because these > resources are donated by the cloud provider? > That's my best guess to justify this. Yes, resources are donated by the cloud providers. I think providers tend to use special flavors to control how are resources are scheduled. > > > for something not load-balanced that requires production downtime, it's a > > very manual process > > Is it also the case that the load-balancing settings are not recorded > anywhere? > eg minimum and maximum number of machines in a load-balancing cluster, > machine flavor Our load balancers don't currently do auto scaling. But configs do live in config management. For example: https://opendev.org/opendev/system-config/src/branch/master/playbooks/group_vars/gitea-lb.yaml There are longer term plans to host that service in kubernetes which could make use of scaling, but gitea the software isn't capable of running in a true cluster yet. > > > It's just a trade-off between development time to create and maintain > > that, and how often we actually start control-plane servers. > > Are the control-plane servers the only cloud cost aspect to outweigh > the development costs? > I'm surprised there isn't already a tool out there that interfaces with > rrdtool and/or cacti to help with this. > rrdtool seems to have been around since 20 years now [1] [2] The expectation from cacti is likely that you'll use the same snmp MIB data that cacti uses to build its graphs to query sizing info directly if you want it. Rather than expecting cacti or rrdtool to expose that to you. > > [1] https://tobi.oetiker.ch/webtools/appreciators.txt > [2] > https://github.com/oetiker/rrdtool-1.x/commit/37fc663811528ddf3ded4fe236ea26f4f76fa32d#diff-dee0aab09da2b4d69b6722a85037700d > -- > Shadi Akiki > Founder & CEO, AutofitCloud > https://autofitcloud.com/ > +1 813 579 4935 ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] opensource infra: server sizes
On Mon, Aug 12, 2019, at 9:43 AM, Shadi Akiki wrote: > Hello. I've been going through the opensource infrastructure > configurations at https://docs.openstack.org/infra/system-config/ > and linked from https://opensourceinfra.org/ > > I don't see any information about server sizes. > Is this something that is not interesting to share as part of the > opensource infrastructure initiative? Our test node sizing is documented here [0]. This has remained fairly static as we try to ensure the test sizes are small enough that developers have the opportunity to replicate results locally without needing a datacenter. For the control plane those server sizes tend not to remain fixed over time as demand rises and falls. You can see a current snapshot of server sizing on our cacti server [1]. These values may change as we find the servers are too small or too big though. [0] https://docs.openstack.org/infra/manual/testing.html [1] http://cacti.openstack.org/cacti/graph_view.php Hope this helps, Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Weekly Infra Team Meeting Agenda for August 6, 2019
We will be meeting tomorrow at 19:00 UTC in #openstack-meeting on freenode with this agenda: == Agenda for next meeting == * Announcements ** clarkb busy with other meetings August 13, 2019. Will need volunteer meeting chair. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Still have OOM issues, though they are less frequent. *** https://etherpad.openstack.org/p/debugging-gitea08-OOM * General topics ** Trusty Upgrade Progress (clarkb 201900806) *** Testing wiki-dev02 *** Next steps for hosting job logs in swift ** Cloud status update (clarkb 201900806) *** FortNebula status update *** Linaro and MOC updates ** AFS mirroring status (clarkb 20190806) *** Fedora did not update for about a month. Cleanup in progress as well as reducing total size of volume (should make vos release more reliable) *** Debian buster updates not populated by reprepro but are assumed to be present by our mirror setup roles. ** PTG Planning (clarkb 20190806) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Agenda for Team Meeting on July 30, 2019 at 19:00UTC
== Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Rebuilding Gitea servers to add disk space and swap. * General topics ** Trusty Upgrade Progress (clarkb 20190730) *** wiki-dev02 puppetry now works? *** Zuul log rendering now ready for review: https://review.opendev.org/#/c/673105/ and parents Would allow us to switch to swift hosted logs and delete the log server. ** Cloud status update (clarkb 20190730) *** FortNebula status update *** Linaro and MOC updates ** PTG Planning (clarkb 20190730) *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Team meeting agenda for July 23, 2019
Our next meeting will be July 23, 2019 at 19:00UTC. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Rebuilding Gitea servers to add disk space and swap. * General topics ** Trusty Upgrade Progress (clarkb 20190723) ** Cloud status update (clarkb 20190723) *** FortNebula status update *** Linaro and MOC updates ** PTG Planning (clarkb 20190716) *** Surveys submitted for OpenDev and Gitea *** https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Issue on instance launching
On Tue, Jul 16, 2019, at 7:07 PM, Vasanth M.Vasanth wrote:
> Hi all,
>
> When I try to start the nova compute service as conflict the ports to
> metadata service. I have killed/stop the process but again listen the
> port and throw the error . Unable to start the both service .
>
> Could you please suggest me.
We are the team that manages the developer infrastructure for building
openstack the software, but don't have much experience running openstack
directly ourselves. You may have better luck with this request sending it to
openstack-disc...@lists.openstack.org which is the general openstack list and
has developers, operators, and users subscribed to it.
>
> openstack-nova-os-compute-api.service
> openstack-nova-metadata-api.service
>
> Nova Api logs ;
>
>
> 2019-07-16 18:40:04.236 31644 WARNING
> oslo_reports.guru_meditation_report [-] Guru meditation now registers
> SIGUSR1 and SIGUSR2 by default for
> backward compatibility. SIGUSR1 will no longer be registered in a
> future release, so please use SIGUSR2 to generate reports.
> 2019-07-16 18:40:04.238 31644 DEBUG nova.wsgi [-] Loading app metadata
> from /etc/nova/api-paste.ini load_app
> /usr/lib/python2.7/site-packages/nova/wsgi.py:496
> 2019-07-16 18:40:04.257 31644 ERROR nova.wsgi [-] Could not bind to
> 0.0.0.0:8775
> 2019-07-16 18:40:04.258 31644 CRITICAL nova [-] error: [Errno 98]
> Address already in use
> 2019-07-16 18:40:04.258 31644 ERROR nova Traceback (most recent call
> last):
> 2019-07-16 18:40:04.258 31644 ERROR nova File
> "/usr/bin/nova-api-metadata", line 10, in
> 2019-07-16 18:40:04.258 31644 ERROR nova sys.exit(main())
> 2019-07-16 18:40:04.258 31644 ERROR nova File
> "/usr/lib/python2.7/site-packages/nova/cmd/api_metadata.py", line 50,
> in main
> 2019-07-16 18:40:04.258 31644 ERROR nova server =
> service.WSGIService('metadata', use_ssl=should_use_ssl)
> 2019-07-16 18:40:04.258 31644 ERROR nova File
> "/usr/lib/python2.7/site-packages/nova/service.py", line 311, in
> __init__
> 2019-07-16 18:40:04.258 31644 ERROR nova max_url_len=max_url_len)
> 2019-07-16 18:40:04.258 31644 ERROR nova File
> "/usr/lib/python2.7/site-packages/nova/wsgi.py", line 101, in __init__
> 2019-07-16 18:40:04.258 31644 ERROR nova self._socket =
> eventlet.listen(bind_addr, family, backlog=backlog)
> 2019-07-16 18:40:04.258 31644 ERROR nova File
> "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 43, in
> listen
> 2019-07-16 18:40:04.258 31644 ERROR nova sock.bind(addr)
> 2019-07-16 18:40:04.258 31644 ERROR nova File
> "/usr/lib64/python2.7/socket.py", line 224, in meth
> 2019-07-16 18:40:04.258 31644 ERROR nova return
> getattr(self._sock,name)(*args)
> 2019-07-16 18:40:04.258 31644 ERROR nova error: [Errno 98] Address
> already in use
> 2019-07-16 18:40:04.258 31644 ERROR
>
The above having been said the problem appears to be that the address is
already in use. You should check netstat or ss to see what is listening on tcp
port 8775 and kill that before starting the service again.
>
> Thanks
> __Vasanth
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for July 9, 2019
== Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Trusty Upgrade Progress (clarkb 20190709) ** Mirror setup updates (clarkb 20190709) *** Do we replace existing mirrors with new opendev mirrors running openafs 1.8.3? ** Cloud status update (clarkb 20190709) *** FortNebula addition * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for July 2, 2019
== Agenda for next meeting == * Announcements ** July 4 is major US holiday. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Trusty Upgrade Progress (clarkb 20190701) ** https mirror update (clarkb 20190701) *** OpenAFS 1.8.3 (via infra PPA) is stable on bionic *** New mirrors should be built with Bionic and OpenAFS 1.8.3 via PPA ** Cloud status update (clarkb 20190702) *** FortNebula addition *** Limestone networking *** Inap upgrades * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for June 25, 2019
== Agenda for next meeting == * Announcements ** Zuul Cloner shim and Bindep fallback file removed from base OpenDev jobs * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Trusty Upgrade Progress (clarkb 20190625) ** https mirror update (clarkb 20190625) *** AFS on Bionic (kafs vs openafs) *** Status update *** https://review.opendev.org/#/q/status:open+branch:master+topic:kafs * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] getting some advanced stats out of gerrit reviews
On Thu, Jun 20, 2019, at 2:15 AM, Sorin Sbarnea wrote: > I would like to build some stats related to openstack gerrit reviews > and I would like to know few things: > * how can I get a dump of the reviews with all comments? (queries on > live DB are not an option) You can query for the list of changes using https://review.opendev.org/Documentation/rest-api-changes.html#list-changes. The query can include a start and stop date too. Then iterate over that list of changes and get their comments with https://review.opendev.org/Documentation/rest-api-changes.html#list-change-comments. > * did anyone already invested time in this direction? (happy to team-up > instead of having a DIY) I believe ttx has done similar work to find different stats in the past. I want to say one of the big take aways from that was to not query all changes every time. You want to run periodically then query only the changes that have updated since your last run. > > One graph I do want to build is would be with average number of > rechecks done on reviews before they are merged, per month. As you can > see that is not a trivial query to do but is doable. > > Another metric it would be fun to build is the estimated failure-rate > for a single CR. As I can compute an average failure rase for each job, > it should not be hard to compute an estimate failure rate when I have a > the list of jobs triggered for specific change. That second one may not > be very practical but I am really curious what kind of values I would > endup with (obvious the range would vary a lot between projects). You may be able to approximate this simply by checking graphite for failure rates on jobs then assume failures are independent (this assumption likely untrue but probably good enough for a start) then calculate the aggregate chance of failures for those jobs. > > Thanks > Sorin ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for June 11, 2019
== Agenda for next meeting == * Announcements ** Clarkb Out June 17-20. Need volunteer to run the meeting June 18. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Trusty Upgrade Progress (clarkb 20190611) ** https mirror update (clarkb 20190611) *** Ready to deploy across all regions? Seems like the ubuntu mirror issues have subsided? ** GitHub replication (corvus 20190611) *** http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.html ** Requesting Spamhaus PBL exceptions (fungi 20190611) ** Replacing SSL certs this week (clarkb 20190611) * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] SPF enforcement on mailing lists
On Fri, Jun 7, 2019, at 2:51 PM, Jeremy Stanley wrote: > This is just a quick heads up that, in order to deal with recent > spam flooding from spoofed E-mail addresses claiming to originate > from domains which provide strict IETF RFC 7208 SPF records[*], we > have instituted a change to reject messages failing SPF "-all" > policies for their domains (not those with "?all" or "~all") at time > of receipt to our listservs. If anyone encounters issues which look > like a rejection resulting from this change in behavior, please > reach out to us in the #openstack-infra channel on the Freenode IRC > network. > > [*] https://tools.ietf.org/html/rfc7208 > -- > Jeremy Stanley And to confirm that ?all isn't rejected, here is an email from a domain with an ?all in its spf record. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for June 04, 2019
== Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Trusty Upgrade Progress (clarkb 20190604) ** https mirror update (clarkb 20190604) *** Ready to deploy across all regions? ** ARM64 Nodepool builder status (clarkb 20190604) * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Problems cloning repos from opendev.org in CentOS7
On Fri, May 24, 2019, at 12:17 PM, Alfredo Moralejo Alonso wrote: > > > On Fri, May 24, 2019 at 9:00 PM Jeremy Stanley wrote: > > On 2019-05-24 20:41:58 +0200 (+0200), Alfredo Moralejo Alonso wrote: > > > On Fri, May 24, 2019 at 8:33 PM Alfredo Moralejo Alonso > > > > > wrote: > > [...] > > > > For periodic jobs we run every 4 hours i see: > > > > - Last successful jobs started at 23-May-2019 21:06:42 UTC > > > > - First failed jobs with this issue started at 24-May-2019 03:06:37 UTC > > > > > > We just got one that failed and sarted at 24-May-2019 02:25:00 UTC > > [...] > > > > Thanks, between the timing and symptoms, we're starting to suspect > > this could be related to how older Git clients' requests are being > > distributed between our load-balanced backends with inconsistent > > packing. We're weighing some ideas for how to improve things there, > > so stay tuned. The good news is that within the next 24 hours the > > backends will be mostly in sync most of the time, so the symptoms > > you're observing may subside for the most part once that's the case > > (but to be entirely honest, we're still not quite sure just yet). > > ok, thanks for the update. I'll switch some jobs to use github mirrors > as workaround until we find it more stable. Can you test opendev.org again? We've updated the load balancer method from least connections to a source IP hash. We think this will address the state mismatch problems that you noticed previously. I have since been able to run a git clone --mirror and git fetch origin --prune myself (when it didn't work reliably in the past). Please let us know if it works now or if it doesn't. That data will be useful either way. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Infra meeting agenda for May 28, 2019
Sorry this is getting out late this week. Yesterday was a holiday in my corner of the world and I didn't end up getting in front of the keyboard. == Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:update-cfg-mgmt *** Puppet 4 effectively done. arm64 hosts running puppet 3 due to lack of packages for puppet 4 on arm64. Affects mirror and builder node. Our backup host which is in the disabled group is also puppet 3 but not running config mgmt. *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Project renames to clean things up after the great bit OpenDev rename (clarkb 20190528) *** Friday May 31, 2019 *** https://review.opendev.org/#/q/topic:project-rename+status:open ** Trusty Upgrade Progress (clarkb 20190528) ** https mirror update (clarkb 20190528) *** https://review.opendev.org/#/c/661187/ switch rax dfw mirror to new opendev mirror. * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Problems cloning repos from opendev.org in CentOS7
On Fri, May 24, 2019, at 10:23 AM, Alfredo Moralejo Alonso wrote: > Hi, > > Since last hours puppet-openstack-integrating jobs in RDO are failing > to deploy puppet modules from > https://opendev.org/openstack/puppet- in CentOS7. > > The sympthom is jobs get stuck randomly when installing modules using > r10k. Investigating it, I've been able to reproduce the issue running: > > git clone --mirror https://github.com/openstack/puppet- > > About 50% of the times, commands get stuck for any project in > opendev.org/openstack. I've checked that it works fine if i don't use > "--mirror". > > As it seems to work fine with fedora, I've compiled and installed > latest version of git from upstream and in a CentOS7 and then, git > clone --mirror works fine, but in this case r10k also gets stuck when > running: > > git --git-dir > /home/centos/.r10k/git/https---opendev.org-openstack-puppet-keystone > fetch origin --prune > > In this case, it only happens with puppet-keystone. > > Note that it always works fine using github.com/openstack mirrors > instead of opendev.org so the problem seems to be some combination of > git repos from opendev with CentOS7 git. Has there been any change that > may be affecting git repos behavior somehow? Yesterday we replicated all of refs/changes and refs/notes from Gerrit to Gitea. We had these replicated on the old Cgit farm, but gitea didn't support this replication until wednesday when we upgraded Gitea. The manpage for git clone --mirror says it will "map all refs (including remote-tracking branches, notes etc.)" so this is possibly related. When you say it gets stuck randomly does that mean the git process never completes the clone? or it stops with failure? In either case stracing the clone process might help to figure out if it is these new refs that it is having problems with. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for 1900UTC May 21, 2019
== Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:puppet-4 and topic:update-cfg-mgmt *** We now install puppet-4 directly using the puppet-agent package in puppet inc's archive as they deleted the proper apt repo. *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Project renames to clean things up after the great bit OpenDev rename (clarkb 20190521) *** Friday May 31, 2019 *** https://review.opendev.org/#/c/655476/ Fixes for rename playbook *** Need changes for infra projects that were not renamed properly *** Need changes for openstack-dev projects that were not renamed properly. ** Trusty Upgrade Progress (clarkb 20190521) *** Progress with askbot thanks to ianw ** https mirror update (clarkb 20190521) *** https://review.opendev.org/658281 -- the actual change to implement opendev.org mirrors * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Airship project replication/sync to GitHub
On Thu, May 16, 2019, at 8:06 AM, Clark Boylan wrote: > On Wed, May 15, 2019, at 4:17 PM, SKELS, KASPARS wrote: > > > > Hi OpenStack infra team, > > > > > > we would like to set up replication > > > > From https://opendev.org/airship -> https://github.com/airshipit > > > > > > The GitHub space airshipit should already be available to OpenStack > > Foundation. > > > > Let us know how to proceed with this. > > > > > > Kindly, Kaspars & Airship team > > The first step is for us to migrate the Airship repos in github under > https://github.com/openstack to https://github.com/airshipit. To do > this you will need to add the "OpenStack Infrastructure" openstackadmin > account as admin on the airshipit org (this is a necessary permission > to do transfer between orgs). Then provide us with a list of the repos > that should be moved and we'll run our script over them to perform the > org transfer. > > This org transfer ensures that github redirects are created so that > people using the older urls will find the new urls. Once that is done > you can remove the openstackadmin account from the airshipit org. > > Then the next step is to set up the Zuul git replication jobs as > described at > http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.html > for each of these repos (this is something you can do in your repos without > infra involvement). > > To get things going maybe respond to this email thread with the list of > repos to transfer on github once the openstackadmin account is added to > the airshipit org? As a followup we successfully added the openstackadmin account to the github org, transferred 16 repos, and then removed openstackadmin from the org. Airship is now in a position to self manage the replication from opendev (via zuul jobs) as well as self manage the Github org. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Airship project replication/sync to GitHub
On Wed, May 15, 2019, at 4:17 PM, SKELS, KASPARS wrote: > > Hi OpenStack infra team, > > > we would like to set up replication > > From https://opendev.org/airship -> https://github.com/airshipit > > > The GitHub space airshipit should already be available to OpenStack > Foundation. > > Let us know how to proceed with this. > > > Kindly, Kaspars & Airship team The first step is for us to migrate the Airship repos in github under https://github.com/openstack to https://github.com/airshipit. To do this you will need to add the "OpenStack Infrastructure" openstackadmin account as admin on the airshipit org (this is a necessary permission to do transfer between orgs). Then provide us with a list of the repos that should be moved and we'll run our script over them to perform the org transfer. This org transfer ensures that github redirects are created so that people using the older urls will find the new urls. Once that is done you can remove the openstackadmin account from the airshipit org. Then the next step is to set up the Zuul git replication jobs as described at http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.html for each of these repos (this is something you can do in your repos without infra involvement). To get things going maybe respond to this email thread with the list of repos to transfer on github once the openstackadmin account is added to the airshipit org? Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] 2019 Denver Summit/Forum/PTG Recap
On Tue, May 14, 2019, at 11:41 AM, Jeremy Stanley wrote: > On 2019-05-13 13:49:44 -0400 (-0400), Clark Boylan wrote: > [...] > > we discussed if OpenDev would be an appropriate location for > > throwaway repos and unfortunately we don't think we are there yet. > > Gerrit currently won't scale to that use case for us. > [...] > > Thinking about this a little more, what if we allowed personal > sandboxes in Gerrit and just didn't wrap them in all the other > replication, CI and general formality we do for normal repositories? > Gerrit will allow you to grant access to paths with "$user" in > them[*], of which WMF's[**] and some other deployments take > advantage to that end. These are ref paths not repo paths. Are you thinking every user could have a single scratch space repo then have arbitrary refs within that repo? > > [*] > https://gerrit-review.googlesource.com/Documentation/access-control.html#_project_access_control_lists > [**] https://www.mediawiki.org/wiki/Gerrit/personal_sandbox > -- > Jeremy Stanley ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Infra Meeting Agenda for May 14, 2019
== Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:puppet-4 and topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Next steps * General topics ** Trusty Upgrade Progress (clarkb 20190507) ** https mirror update (ianw 20190514) * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] 2019 Denver Summit/Forum/PTG Recap
Apologies for not getting this out earlier, but last week was a very odd week for. Slowly returning to normal now though. The long week started with a board meeting. In that board meeting Zuul was confirmed as a top level OpenStack Foundation Project. An exciting start to the week for our friends in Zuul land (which happens to be many of us too)! The first day of the summit we had Keynotes where corvus talked about the way we speculatively test changes with containers. ttx had a talk after keynotes on "Open Source is not enough" [0] which helps to point out why the infrastructure that we run is valuable. Later that day we had a Forum session on Storyboard painpoints. There was good feedback and we got a couple volunteers to help fix existing issues [1]. Tuesday started with a back to back Zuul BoF then Working Group session (in the same room) that ended up being a good discussion among current and potential Zuul users. We discussed how people were using Zuul and hangups upgrading from v2 to v3. There was a forum discussion on PTL tips and tricks [2] which I found interesting (as current PTL). In particular is seems that a number of projects use regular update emails (like Zuul) to keep people up to date on goings on rather than the weekly meeting. We recently discussed this as a team so I don't think we need to rehash it but was good to see other approaches. This day ended with a session on improving the Help Most Wanted list [3] which we are still on. General consensus seemed to be that the first pass at writing business cases addressed the wrong audience so a second pass would be made. In addition to that Allison Randall would reach out to institutions like OSUOSL to see if there is possibility to collaboration there. The last day of the summit included a forum session on OpenLab [4]. This was useful to help communicate where OpenDev can serve its users and where OpenLab is able to help too. At this point I think I was a zombie and we finished up the Summit with back to back Infra and Zuul project updates. But wait there is more! We had an Infra PTG room the next two days. Notes were taken at this etherpad [5] under the bolded headings. We kicked things off with a rundown of recent tech changes so that everyone had a rough idea of what those looked like. Jhesketh showed us that docker-compose can do things like get logs for your composed docker containers :) We discussed what the future of our deployment tooling looks like using Zuul as a CD engine and how we transition to that point. The current plan is that we'll split up the base.yaml playbook into other playbooks (but continue to have cron drive them). Then we can have Zuul take over the execution of each playbook over time (and possibly even use Zuul's periodic pipeline to do convergence if things are skipped for some reason). Initial work on that has started here [6]. We also talked about what new repo and namespace creation looks like in OpenDev. We ended up with a plan that roughly comes down to individual namespaces will be self managed as much as possible. This means that the Infra team won't be approving what goes into openstack/ or airship/ or zuul/ instead appropriate representatives from those groups will make those approvals. To make this happen we'll likely end up with a metadata repo per namespaces that tracks who can approve these things then rely on some combo of Gerrit config and Zuul jobs to make that happen. We also discussed how the namespaces might manage info needed to create valid zuul main.yaml then we can have tooling aggregate that together and deploy it. This way we don't have to manage each of those additions for zuul directly. Finally we discussed if OpenDev would be an appropriate location for throwaway repos and unfortunately we don't think we are there yet. Gerrit currently won't scale to that use case for us. Last major item we poked at was improving the reliability of our docker based jobs, particularly those that run on ipv6 clouds. The plan there is to use the in region mirror nodes for dockerhub proxying rather than local buildset registry proxies as the in region mirror has a floating IP to do 1:1 ipv4 NAT but the test nodes are many:1 ipv4 NAT. We expect this will improve reliability significantly. Before we can do that though we need to enable TLS on those mirror nodes as we'll be publishing the resulting artifacts. We can do this via letsencrypting new opendev.org mirror endpoints. The StarlingX folks came over to talk to us about their testing needs. They would like to do larger test scenarios on multiple nodes and ideally on baremetal. We suggested that they start by scaling up the testing on VMs (as many issues fall out when you go from one to two to three nodes) and getting that tested even on VMs would be valuable. For the baremetal problem we pointed out that we can support baremetal testing, we just need someone to give us access to
Re: [OpenStack-Infra] Question: How to attach large files to LP bugs?
On Fri, May 10, 2019, at 6:47 AM, Jones, Bruce E wrote: > > Greetings Infra team. The StarlingX project has a question – we have > had some bug reports where the log files needed to debug the bug turn > out to be quite large. Larger than can be attached to a LP entry. > > > Is there a recommended way to handle this case? I'm not even sure what the launchpad attachment size limit is, so I think this is the first time I have had to consider this problem. One option if possible would be to use an efficient compression algorithm like xz to compress the logs down to a size that LP will accept the upload. If that still doesn't get things small enough maybe compress multiple chunks and upload them separately? If we can't come up with a workaround like the above options, then the next best option is likely to host the content somewhere else and link LP to that location. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Move packstack from openstack to redhat-openstack organization in github
On Tue, Apr 30, 2019, at 7:51 AM, Alfredo Moralejo Alonso wrote: > Hi, > > After migration to opendev an moving openstack/packstack to > x/packstack, mirroring to github has been lost as announced in [1]. > > Could we get the repository moved to redhat-openstack organization in > github?, we'd like to get redirection from old repo to the new one > what's require to transfer the repo using github iiuc. > > Best regards, > > Alfredo > > [1] > http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005629.html > Thank you for sending this out so that we have a record of why the move was made. I have no objections to making this move in GitHub. If dmsimard still has time to do the move I think we can go ahead with that. Otherwise, we'll likely get to it once we return to our daily lives after the summit and PTG. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Low Key Monday Evening Get Together
On Fri, Apr 26, 2019, at 3:19 PM, Clark Boylan wrote: > On Wed, Apr 24, 2019, at 7:40 PM, Wesley Hayutin wrote: > > Just some local input.. > > That is a long way to go from downtown. There are plenty of outdoor > > venues with great beer a little closer. You might want to try a > > personal favorite [1], there are ton of other breweries around odell as > > well. Hard to go wrong there. Either way, enjoy your time in Denver! > > > > [1] https://www.odellbrewing.com/taproom-denver/ > > Yup, I know it is a bit of a trek, but we've gone to Lowry's each of > the last two PTGs in Denver. It would be great to get back. That said > the forecast for Monday night/Tuesday morning continues to look less > and less pleasant so I think for Monday night team activities our best > bet may be to stick closer to the event venue and be indoors. > > As I mentioned before I'll send concrete details once onsite, but I > would expect we go to FTW Denver at this point. Zuul has been confirmed. Given that, I'm more than happy to have an Infra + Zuul team gathering Monday evening. We can celebrate Gertty's birthday and Zuul's confirmation :) Let's meet in front of the big blue bear statue at the convention center at 6:45pm Monday evening. From there we'll go to FTW Denver. Unfortunately, the weather calls for heavy rain and possibly snow Monday night so Lowry seems like a bad choice Monday. I'm told that we can wait inside of the convention center near the bear (it is public space) which is probably a good idea if it is pouring rain. Looking forward to seeing you there! Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Low Key Monday Evening Get Together
On Wed, Apr 24, 2019, at 7:40 PM, Wesley Hayutin wrote: > Just some local input.. > That is a long way to go from downtown. There are plenty of outdoor > venues with great beer a little closer. You might want to try a > personal favorite [1], there are ton of other breweries around odell as > well. Hard to go wrong there. Either way, enjoy your time in Denver! > > [1] https://www.odellbrewing.com/taproom-denver/ Yup, I know it is a bit of a trek, but we've gone to Lowry's each of the last two PTGs in Denver. It would be great to get back. That said the forecast for Monday night/Tuesday morning continues to look less and less pleasant so I think for Monday night team activities our best bet may be to stick closer to the event venue and be indoors. As I mentioned before I'll send concrete details once onsite, but I would expect we go to FTW Denver at this point. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Low Key Monday Evening Get Together
On Wed, Apr 24, 2019, at 9:14 AM, Clark Boylan wrote: > Hello Infra! > > Monday evening is looking like a good night to have a low key informal > team gathering. The only official event on the calendar is the > Marketplace Mixer which runs until 7pm. Weather permitting I'd like to > head back to Lowry Beer Garden (where we've gone at past PTGs). It is a > bit out of the way from the conference center so we will need to > coordinate uber/lyft transport but that hasn't been a problem in the > past. > > Lets meet at 6:30pm (I can send specific location once onsite) and head > over to Lowry. If the weather looks terrible I can call ahead and see > if their indoor area is open and if they say it is "meh" we'll find > something closer to the conference center. Also, they close at 9pm > which should force us to get some sleep :) > > Finally, Monday is a good day because it is gertty's birthday. Hope to > see you then. Hey, it was pointed out that there is a really good chance that Zuul will be confirmed as a top level OSF project at the board meeting on Sunday. This seems like an excellent thing to celebrate as well, and given the overlap between Zuul and Infra teams I think we should celebrate that together. Don’t want to put the cart before the horse though, so we can wait until we know for sure before we put a label on it. But please go ahead and mark Monday evening for Zuul too just in case. Also looking at the weather more closely it will be about 50F/11C with rain and wind Monday evening, so Lowry may not be the best choice. It has been pointed out that For The Win Denver would be willing to take us as an informal group and we can play arcade games indoors instead. They have food and drinks too. As things firm up I'll send an update to this email with concrete time and location details. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Low Key Monday Evening Get Together
On Wed, Apr 24, 2019, at 9:14 AM, Clark Boylan wrote: > Hello Infra! > > Monday evening is looking like a good night to have a low key informal > team gathering. The only official event on the calendar is the > Marketplace Mixer which runs until 7pm. Weather permitting I'd like to > head back to Lowry Beer Garden (where we've gone at past PTGs). It is a > bit out of the way from the conference center so we will need to > coordinate uber/lyft transport but that hasn't been a problem in the > past. > > Lets meet at 6:30pm (I can send specific location once onsite) and head > over to Lowry. If the weather looks terrible I can call ahead and see > if their indoor area is open and if they say it is "meh" we'll find > something closer to the conference center. Also, they close at 9pm > which should force us to get some sleep :) > > Finally, Monday is a good day because it is gertty's birthday. Hope to > see you then. > Also Tuesday night is the official Party event. Sign up at https://www.eventbrite.com/e/the-denver-party-during-open-infrastructure-summit-tickets-58863817262 with password "denver". If you are like me and don't want to give out a phone number just enter "555-555-" in that field. These details apparently went out to attendees already but I didn't get said email so posting it here just in case it is useful for others too. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] [Starlingx-discuss] StarlingX / Infra teams sync-up at PTG
On Tue, Apr 23, 2019, at 4:55 PM, Saul Wold wrote: > > > Infra Team, > > The StarlingX Project would like to request a 30-60 minute sync up to > talk about some of the challenges that StarlingX faces with regards to > build and test infrastructure. > > As StarlingX is an integration project that creates a Linux Distribution > with a Cloud infrastructure on top of it, this makes it more challenging > to both build and test. The current OpenStack Foundation infrastructure > is good as building and testing projects such as Nova, Neutron, ... It > could be used to build and test the individual components of the > StarlingX Flock, such as Fault and others. It's not well suited to build > the complete StarlingX ISO and test that ISO. > > We want to explore what the existing resources that are available and > understand how and what we can add to the infrastructure to enable the > build and testing that StarlingX will require. > > We hope that we can find an hour timeslot during the PTG that we can > talk further about this with both teams. Our timeslots overlap Thrusday > afternoon and Friday so we could put it on our adgendas. > > Please let us know if you have available time slots. I'm thinking the best day for us is Friday (we'll want to dig into our team specific items on Thursday). Does Just after the lunch break on Friday (say 1:30pm) in the Infra/QA room work? If so we'll see you there. I'll go ahead and add you to our etherpad [0] as I'm editing that today too. [0] https://etherpad.openstack.org/p/2019-denver-ptg-infra-planning Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for April 23, 2019
== Agenda for next meeting == * Announcements ** Summit and PTG next week. No meeting April 30. * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:puppet-4 and topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** Catch up on what needs to be done now that we've migrated. * General topics ** LetsEncrypt Progress (clarkb 20190423) *** git.airshipit.org, git.starlingx.io and survey.openstack.org expire soon. Harder to LE these due to where dns is hosted. Should we buy 2 year certs and then call it a day? ** Trusty Upgrade Progress (clarkb 20190423) ** PTG planning (clarkb 20190423) *** https://etherpad.openstack.org/2019-denver-ptg-infra-planning * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Post opendev git repo hosting status repo
On Fri, Apr 19, 2019, at 9:18 PM, Clark Boylan wrote: > Hello infra! > > We've completed the initial work to do this migration. YAY. But there > are a few things to keep in mind while we clean up after ourselves. > > This was a bit of an inception moment where we rely on Gerrit to drive > our infrastructure but we renamed and moved Gerrit. This means that the > ansible crons on bridge.o.o are disabled. Please DO NOT reenable the > run_all.sh cron until we are ready. > > We should also avoid creating new projects until we are happy with > things and have run_all.sh running again. config-core please DO NOT > merge new project additions until we are ready. > > In general though bug fixes to zuul jobs should be fine (and I expect > we'll have to start here in order to merge other code). And we'll fix > more bugs from there. Thanks again to everyone that helped and for your > patience otherwise. This was the work of many people. > > Also I'll respond to this when we think we are ready to go on the > things I asked us to leave alone for now. We have reenabled the run_all.sh cron on bridge.openstack.org after iterating through each playbook it runs and examining the results. Things looked happy so we are back to operating with normal deployment procedures. That said we would like to keep a freeze on new project creations for a little while longer. We found some bugs around creating and renaming projects particularly if the new target org (or namespace) does not already exist. This means config-core and infra-root please avoid creating new projects until we are ready. I will followup to this message with a note when we have reached that point. Thank you, Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Post opendev git repo hosting status repo
Hello infra! We've completed the initial work to do this migration. YAY. But there are a few things to keep in mind while we clean up after ourselves. This was a bit of an inception moment where we rely on Gerrit to drive our infrastructure but we renamed and moved Gerrit. This means that the ansible crons on bridge.o.o are disabled. Please DO NOT reenable the run_all.sh cron until we are ready. We should also avoid creating new projects until we are happy with things and have run_all.sh running again. config-core please DO NOT merge new project additions until we are ready. In general though bug fixes to zuul jobs should be fine (and I expect we'll have to start here in order to merge other code). And we'll fix more bugs from there. Thanks again to everyone that helped and for your patience otherwise. This was the work of many people. Also I'll respond to this when we think we are ready to go on the things I asked us to leave alone for now. With that, its been a really long day, and I should get some rest. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] OpenDev git hosting migration and Gerrit downtime April 19, 2019
On Wed, Apr 17, 2019, at 7:54 AM, Thierry Carrez wrote: > James E. Blair wrote: > > My understanding is this is due to openstack-infra being TC-governed and > > opendev not quite having gotten around to establishing an official > > non-TC governance yet. I think the intent is to eventually do that. We > > could probably anticipate that a bit if we would like and go ahead and > > sort openstack-infra things into different buckets. At the end of this, > > I think we will all have more hats, with overlap between opendev and > > openstack. Some current infra activities and repos are > > openstack-specific and should be re-homed into openstack; others serve > > all projects and should be in opendev; yet more are just things that are > > incidentally related to what we do and should be on their own. > > > > I've produced a list based on my estimation of what things will look > > like at the end of the process. This is just a starting point if we > > would like to explore this option. We could refine the list and use it, > > or we could choose to stick with the status quo temporarily and move the > > infra repos out of openstack at a later time when things are more clear. > > > > https://etherpad.openstack.org/p/6CmVhW40m0 > > I understand it conflates two separate issues (rename and governance), > and we clearly don't want to delay the rename due to premature > governance discussions. > > My personal view on this is that we can split openstack-infra things > between openstack/ and opendev/ following your etherpad plan, and > officialize the governance part later. Technically, things split under > opendev/ would still be under the OpenStack infra team and the OpenStack > TC until the governance is officially split out. That avoids unnecessary > renames and crowding our new clean openstack/ space with stuff we know > will move away soon. > > We'll obviously have to edit the projects.yaml to account for the repo > renames under the Infrastructure team, but we have to do that anyway. This plan works for me. I did note a few items in the x/ namespace that probably do make sense under openstack/. We'll also need to make a decision on storyboard. Any suggestions for storyboard? fungi in particular may have thoughts on that. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Offering help / sysadmin background with heavy emphasis on systems automation (Ansible)
On Mon, Apr 15, 2019, at 11:27 PM, Mihai Rusan wrote: > Hello, > > I'm Mihai, over 5+ years of sysadmin background and embracing Ansible > and cloud infrastructure. > > Happy to help, I'm living in Romania, Europe. Time zone EEST UTC+3. Hello and thank you for the offer to help. This week is a bit busy for us as we are planning a large git repository hosting migration, apologies if responses are a bit slow. That said we are in the middle of a large transition from puppet based deployments to ansible and docker based deployments. We have written up a specification for this work which can be found at https://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html. I mention this as you mentioned ansible so I thought it might be interesting to you. The other specifications on that site should give you an idea of the other work we have in progress as well as other items may interest you too. If that spec or others look interesting feel free to reach out to myself or this list and we can take it from there. You may also want to hop into #openstack-infra on freenode's IRC servers and say hello as well as listen in as we discuss things. That should give you a good flavor for the day to day efforts. The last thing I'll mention is that a good portion of our day to day efforts involve reviewing code changes to our configuration management tooling. https://review.openstack.org/#/q/project:openstack-infra/system-config+status:open will list out a good portion of those changes if you'd like to take a look and see what that entails. Feel free to do a review or two as well :) Hope this helps, Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Freezing new project creation in Gerrit
Hello config-core reviewers, We'd like to freeze new project creation through the end of the week so that we can get through the OpenDev git transition without a constantly changing list of projects that may be renamed. This will allow us to review the lists and be confident they are static prior to the transition. So with that in mind please hold back from approving new projects in project-config/gerrit/projects.yaml until we are clear of the transition. Thank you, Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for April 16, 2019
== Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:puppet-4 and topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** https://storyboard.openstack.org/#!/story/2004627 *** Setting specific time for maintenance work to start *** Are we ready? What do we need to do between now and Friday? *** Should we include the one requested repo rename below in this process or do that later? * General topics ** LetsEncrypt Progress (clarkb 20190416) *** graphite.opendev.org is now LetsEncypted. Next steps are restarting services when certs update and testing of that process. ** Trusty Upgrade Progress (clarkb 20190416) *** lists.openstack.org has been upgraded. ** Removal of ugo+rw chmod from default job definitions (zbr 20190415) *** Mainly this insanely unsafe chmod does prevent ansible from loading ansible.cfg files from our repos. That is happening from "Change zuul-cloner permissions" task which is run even for basic jobs like openstack-tox-py* ones. * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] lists upgrade seems to be complete
This message acts both as an indication that the upgrade was completed and a sanity check that our lists are functioning. Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting Agenda for April 9, 2019
== Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:puppet-4 and topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** https://storyboard.openstack.org/#!/story/2004627 *** ok to merge git->https changes (ianw 20190409) * General topics ** PTG planning (clarkb 20190409) *** https://etherpad.openstack.org/2019-denver-ptg-infra-planning *** Given condensed schedule do we want to make a schedule unconference style when we get there based on progress on OpenDev and other work or do we want to try and write up a more solid schedule the week before the summit (week after opendev git transition)? ** LetsEncrypt Progress (clarkb 20190409) *** https://review.openstack.org/#/c/636759/ has one +2 ** Trusty Upgrade Progress (clarkb 20190409) *** Thoughts on upgrading lists.openstack.org this Friday the 12th of April? ** Zanata checkin (ianw 20190409) * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Problems building pinned master branch
On Mon, Apr 8, 2019, at 6:14 AM, Thomas Bachman wrote: > > Apparently I'm still learning how to use email (drat those hot-keys!). > > This is the patch: > > https://review.openstack.org/#/c/647475/ > > You can see that the patch initially passed CI on 3/25/2019, but then > when the patch was going to be merged (no changes in the code), the > merge job failed. The way it failed is also interesting -- the error in > the pep8 job indicates that the DB migration script isn't available, > which is part of the python code (i.e. suggesting that our repo's > package isn't even installed?): > > http://logs.openstack.org/75/647475/1/gate/openstack-tox-pep8/5543fd1/job-output.txt.gz#_2019-03-28_23_14_40_267120 > > The py27 job suggests the same: > > http://logs.openstack.org/75/647475/1/gate/openstack-tox-py27/8a13368/job-output.txt.gz#_2019-03-28_23_14_59_087759 > > Does anyone know of any changes between 3/25/2019 and 3/28/2019 that > would result in our package not being included? Again, please let me > know if this is the wrong list for such issues, and thanks in advance > for any/all help! I'm not aware of any changes on our end. There have been a few recent tox releases [0] which may be relevant. When I run the tox -epep8 command locally with tox 3.5.3 I see that '-e git+https://git.openstack.org/openstack/group-based-policy@77a7df0a7093f8c3e245481b237738bef838079d#egg=group_based_policy' shows up, but is not present on the zuul job log. I then upgraded tox to 3.8.6 (latest) and reran it (using the -r flag to reinstall deps) and see the same successful result with that package being installed. As a result I don't think tox is at fault here. I do think your answer is in tracking down why that package isn't installed. Normally tox would install it as a separate last step of the package installation process. [0] https://pypi.org/project/tox/#history Hope this helps, Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Plan for upgrading lists.openstack.org
Hello everyone, Over the past few days I've been working to test that my test lists.openstack.org server (booted from snapshot of the prod server then upgraded in place with do-release-upgrade) works as expected. I think I've got everything nailed down but our plan for dealing with dmarc_moderation_action configuration items in mailman. I've been taking notes on an etherpad [0] if you are curious about the process and what has been tested. Under list item 5.2.6 on the etherpad I have listed out ideas for how we can handle dmarc_moderation_action. I'd like to have a rough plan for that before we upgrade, and I'd like to upgrade soon so any input is much appreciated. If you can please look over that list and give your thoughts on this thread or on the etherpad. [0] https://etherpad.openstack.org/p/lists.o.o-trusty-to-xenial Thank you, Clark ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting agenda for April 2, 2019
== Agenda for next meeting == * Announcements * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:puppet-4 and topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** https://storyboard.openstack.org/#!/story/2004627 * General topics ** PTG planning (clarkb 20190402) *** https://etherpad.openstack.org/2019-denver-ptg-infra-planning *** [https://www.openstack.org/ptg#tab_schedule Draft schedule] Now scheduled for Thursday and Friday. ** LetsEncrypt Progress (clarkb 20190402) ** Trusty Upgrade Progress (clarkb 20190402) *** Need to take the plunge on lists.o.o once mailman is confirmed to work on test node * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Meeting agenda for March 19, 2019
== Agenda for next meeting == * Announcements ** Clarkb on vacation March 25-28 * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:puppet-4 and topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** https://storyboard.openstack.org/#!/story/2004627 *** git:// to https:// translation (ianw 20190312 clarkb 20190319) Clarkb wanted to followup on this and make sure we aren't stuck or confused about what the path forward is. * General topics ** Trusty server upgrade progress (clarkb 20190319) *** https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup *** AFS upgrades complete. Still a few more services remaining. Please grab them on the etherpad if you can do the upgrade. ** PTG planning (clarkb 20190319) *** https://etherpad.openstack.org/2019-denver-ptg-infra-planning *** [https://www.openstack.org/ptg#tab_schedule Draft schedule] We have Friday and Saturday in a shared room with the QA team. * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] OpenStack Infra Team Meeting Agenda for March 12, 2019
== Agenda for next meeting == * Announcements ** PTL Candidate nominations close today (March 12) at 23:45UTC. *** Clarkb has submitted his candidacy for Infra PTL. ** Summit Forum sessions can now be [https://www.openstack.org/summit/denver-2019/call-for-presentations proposed] * Actions from last meeting * Specs approval * Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.) ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/task-tracker.html A Task Tracker for OpenStack] ** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management] *** topic:puppet-4 and topic:update-cfg-mgmt *** Zuul as CD engine ** OpenDev *** https://storyboard.openstack.org/#!/story/2004627 * General topics ** Trusty server upgrade progress (clarkb 20190312) *** https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup *** AFS fileserver upgrades started March 11 (hopefully that is in the future still) * Open discussion ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
