[OpenStack-Infra] [PTG] Zuul interviews at PTG + still seeking a volunteer to talk infra
Per the discussion in last week's Zuul meeting [1] -- I have signed some folks up, in a hopefully-conflicting fashion, to chat about Zuul on camera with Mr. Rich Bowen during the PTG. (One of his previous mails on this topic follows below, for those unfamiliar with what the heck I'm talking about.) This will be lovely content for us to have (a) on the zuul-ci.org page, and (b) to highlight in other places all the great work folks in Infra have been doing. 1: Jim Blair, Jesse Keating -- Fingers crossed, *Monday* at 1:30pm. (Note: Rich isn't officially recording folks on Monday, but is willing to bust out the gear anyway since it's Jesse's only day around. Rich, plz note!) This chat will largely be: What the heck is Zuul, why Ansible, and look, shiny new GitHub integrations that enable not-just-gerrit workflows! 2: Tuesday, 4:30pm: Monty, Ricky, and (hopefully) Tobias. (Tobias: I know you weren't at the irc meeting, but am hoping you're willing to chat on camera :D If not, lmk...) Topic for this one: Why enabling other communities and users with the power of Zuul matters, and look! Examples of folks using it for not-just-openstack! (And I'll be coming along / squirrel herding as well, though not to be on camera. :D) Also: STILL SEEKING a volunteer or two to talk about the other good things Infra is up to. Rich still has spots open, and I will keep poking people until someone volunteers :) I was thinking that a combo of pabelanger + fungi and/or clarkb would be grand (trying to not steal everyone simultaneously and all that jazz.) Feel free to add yourself to the schedule Rich has put together [2]. If folks have questions, or flames / praise for my schedule alignment skills, LMK. Cheers, -robyn [1] http://eavesdrop.openstack.org/meetings/zuul/2018/zuul.2018-02-19-22.01.html [2] https://docs.google.com/spreadsheets/d/1MK7rCgYXCQZP1AgQ0RUiuc-cEXIzW5RuRzz5BWhV4nQ -- Forwarded message -- From: Rich BowenDate: Mon, Feb 19, 2018 at 7:12 AM Subject: [openstack-dev] [PTG] Project interviews at the PTG To: openstack-...@lists.openstack.org I promise this is the last time I'll bug you about this. (Except on-site, of course!) I still have lots and lots of space for team/project/whatever interviews at the PTG. You can sign up at https://docs.google.com/spread sheets/d/1MK7rCgYXCQZP1AgQ0RUiuc-cEXIzW5RuRzz5BWhV4nQ/edit#gid=0 You can see some examples of previous interviews at http://youtube.com/RDOCommunity For the most part, interviews focus on what your team accomplished during the Queens cycle and what you want to work on in Rocky. However, we can also talk about other things like governance, community, related projects, licensing, or anything else that you feel is related to the OpenStack community. I encourage you to talk with your team, and find 2 or 3 people who can speak most eloquently about what you are trying to do, and find a time that works for you. I'll also have the schedules posted on-site, so you can sign up there, if you're still unsure of your schedule. But signing up ahead of time lets me know whether Wednesday is really a vacation day. ;-) See you in Dublin! -- Rich Bowen - rbo...@redhat.com @RDOcommunity // @CentOSProject // @rbowen __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul
On Jun 9, 2017 1:14 PM, "Monty Taylor"wrote: On 06/09/2017 02:35 PM, Clark Boylan wrote: > On Fri, Jun 9, 2017, at 09:22 AM, Monty Taylor wrote: > >> Hey all! >> >> Tristan has recently pushed up some patches related to providing a Web >> Dashboard for Zuul. We have a web app for nodepool. We already have the >> Github webhook receiver which is inbound http. There have been folks who >> have expressed interest in adding active-REST abilities for performing >> actions. AND we have the new websocket-based log streaming. >> >> We're currently using Paste for HTTP serving (which is effectively >> dead), autobahn for websockets and WebOB for request/response processing. >> >> This means that before we get too far down the road, it's probably time >> to pick how we're going to do those things in general. There are 2 >> questions on the table: >> >> * HTTP serving >> * REST framework >> >> They may or may not be related, and one of the options on the table >> implies an answer for both. I'm going to start with the answer I think >> we should pick: >> >> *** tl;dr *** >> >> We should use aiohttp with no extra REST framework. >> >> Meaning: >> >> - aiohttp serving REST and websocket streaming in a scale-out tier >> - talking RPC to the scheduler over gear or zk >> - possible in-process aiohttp endpoints for k8s style health endpoints >> > Hiya: Two things, which may or may not be useful -- some of this is over my head, but I like to at least make sure information that *might* be useful is seen: 1: just read this blog from the nice humans at datadog this morning, which mentions a bunch of things relating to http, logs, events, streaming, protobuf, k8s, etc. -- oh, and the word python, which is nice. :) Obviously the datadog folks have different end goals than we do, but it seems like there might be useful thinking fodder or overlap to consider peeking at. Plus I always like to remember that other tools may want to ingest data in different ways for other purposes, and thinking about that from other points of view can be nice (although not something we necessarily want to block on to get a thing out the door :D) https://engineering.datadoghq.com/protobuf-parsing-in-python/ 2: The Github API is making some changes in moving from V3 to V4. Not clear to me if (a) they will live in coexistence or not, either forever-ish or otherwise, (b) if otherwise, when v3 will be eol'd, but might be worth considering if v4 helps the particular cases discussed in this thread in any way (i dont count "annoying jlk endlessly since he just finished all that github work," *hugs* to jlk tho.) If none of that is useful... sorry for the noise :) -r >> Since we're talking about a web scale-out tier that we should just have >> a single web tier for zuul and nodepool. This continues the thinking >> that nodepool is a component of Zuul. >> > > I'm not sure that this is a great idea. We've already seen that people > have wanted to use nodepool without a Zuul and even without performing > CI. IIRC paul wanted to use it to keep a set of asterisks floating > around for example. We've also seen that people want to use > subcomponents of nodepool to build and manage a set of images for clouds > without making instances. > Excellent point. In the past we have been careful to keep logical tools separate which > has made it easy for us to add new tools and remove old ones. > Operationally this may be perceived as making things more difficult to a > newcomer, but it makes life much much better 3-6 months down the road. > > >> In order to write zuul jobs, end-users must know what node labels are >> available. A zuul client that says "please get me a list of available >> node labels" could make sense to a user. As we get more non-OpenStack >> users, those people may not have any concept that there is a separate >> thing called "nodepool". >> >> *** The MUCH more verbose version *** >> >> I'm now going to outline all of the thoughts and options I've had or >> have heard other people say. It's an extra complete list - there are >> ideas in here you might find silly/bad. But since we're picking a >> direction, I think it's important we consider the options in front of us. >> >> This will cover 3 http serving options: >> >> - WSGI >> - aiohttp >> - gRPC >> >> and 3 REST framework options: >> >> - pecan >> - flask-restplus >> - apistar >> >> ** HTTP Serving ** >> >> WSGI >> >> The WSGI approach is one we're all familiar with and it works with >> pretty much every existing Python REST framework. For us I believe if we >> go this route we'd want to serve it with something like uwsgi and >> Apache. That adds the need for an Apache layer and/or management uwsgi >> process. However, it means we can make use of normal tools we all likely >> know at least to some degree. >> > > FWIW I don't think Apache would be required. uWSGI is a fairly capable > http server aiui. You can also pip install uwsgi so the simple case > remains
[OpenStack-Infra] Status of Zuul v3
Greetings! This periodic update is primarily intended as a way to keep contributors to the OpenStack community apprised of Zuul v3 project status, including future changes and milestones on our way to use in production. Additionally, the numerous existing and future users of Zuul outside of the OpenStack community may find this update useful as a way to track Zuul v3 development status. If "changes are coming in the land of Zuul" is new news to you, please read the section "About Zuul and Zuul v3" towards the end of this email. == Zuul v3 project status and updates == The Big Big news: Updates to Nodepool to support Zuul v3 are done! This has been a large effort (approximately the size of one Stay-Puft marshmallow man), and the team is super excited to now have this in place. Give shrews a high-five if you see him! We still have bugs to shake out, documentation to update, and a backwards-incompatible configuration syntax change to make (http://lists.openstack.org/pipermail/openstack-infra/2017-January/005018.html), but we can consider it feature complete. Continuing to work on and discuss sample jobs: Putting together an ideal "sample job" and documenting what best practices look like is essential in enabling future users of Zuul to quickly and easily create jobs of their own. Paul Belanger has been making progress on this in the form of a generic tox job (see https://review.openstack.org/438281), and we're now at the point of discussing why / how this is a foundational example. Additionally, as more of the major subsystems of Zuul complete their refactoring, the ability for one to contribute to Zuul v3 continues to get a bit easier. Even though at the moment we still recommend having deep familiarity with one or more of the subsystems, we do now have some "low-hanging fruit" tasks listed in storyboard. Use this magical link to find your way: https://storyboard.openstack.org/#!/story/list?status=active=low-hanging-fruit_id=679 Some new specs for enhancements and/or features have been put forth as well: * An interface for Zuul Job Reporting https://review.openstack.org/444088 * Zuulv3 Executor Security Enhancement https://review.openstack.org/95 * Update job trees to graphs https://review.openstack.org/443985 (yes, technically this is an *update* to a spec, not a new one.) Upcoming tasks and focus: * Re-enabling disabled tests: We're continuing to make our way through the list of remaining tests that need enabling. See the list, which includes an annotation as to complexity for each test, here: https://etherpad.openstack.org/p/zuulv3skips * Full task list and plan is in the Zuul v3 storyboard: https://storyboard.openstack.org/#!/board/41 Recent changes: * Zuul v3: https://review.openstack.org/#/q/status:closed+project:openstack-infra/zuul+branch:feature/zuulv3,25 (Early adoptors should be aware we renamed zuul-launcher to zuul-executor, this is a breaking change: https://review.openstack.org/#/c/445594) * Nodepool: https://review.openstack.org/#/q/status:closed+project:openstack-infra/nodepool+branch:feature/zuulv3,25 Previous IRC Meeting minutes & logs: * 2017-03-06 Minutes: http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-03-06-22.03.html * 2017-03-06 Full log: http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-03-06-22.03.log.html * 2017-03-13 Minutes: http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-03-13-22.02.html * 2017-03-13 Full log: http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-03-13-22.02.log.html == About Zuul and Zuul v3 == Zuul is a pipeline-oriented project gating system, driving the project automation necessary to enable a continuous integration environment for the OpenStack community. It directs the OpenStack community’s testing, running tens of thousands of jobs each day, responding to events from the code review system and stacking potential changes to be tested together. Testing and continuous integration for drivers is also enabled by OpenStack’s deployment of Zuul for ~50 third-party projects. Zuul v3 is the upcoming, not-yet-in-production version of Zuul currently under development, bringing (amongst many others) the following improvements and features: * simplifying the ability to run jobs in multi-node environments * improved management of large numbers of jobs and job variations * support for in-tree job configuration * ability to define jobs using Ansible (http://github.com/ansible/ansible) Even though prior versions of Zuul are already in use by numerous projects and companies not related to OpenStack efforts, a primary goal of Zuul v3 is to make Zuul a universally useful CI / CD engine for any size use case. The design of Zuul v3 improves the modularity of the system, and enables new triggers (such as GitHub) and reporters (where results are sent) to be more easily integrated with Zuul; additionally, the ability to execute jobs on non-OpenStack clouds, such as, well, the other ones :D, as well as non-cloud
[openstack-dev] Status of Zuul v3
Greetings! This periodic update is primarily intended as a way to keep contributors to the OpenStack community apprised of Zuul v3 project status, including future changes and milestones on our way to use in production. Additionally, the numerous existing and future users of Zuul outside of the OpenStack community may find this update useful as a way to track Zuul v3 development status. If "changes are coming in the land of Zuul" is new news to you, please read the section "About Zuul and Zuul v3" towards the end of this email. == Zuul v3 project status and updates == The Big Big news: Updates to Nodepool to support Zuul v3 are done! This has been a large effort (approximately the size of one Stay-Puft marshmallow man), and the team is super excited to now have this in place. Give shrews a high-five if you see him! We still have bugs to shake out, documentation to update, and a backwards-incompatible configuration syntax change to make (http://lists.openstack.org/pipermail/openstack-infra/2017-January/005018.html), but we can consider it feature complete. Continuing to work on and discuss sample jobs: Putting together an ideal "sample job" and documenting what best practices look like is essential in enabling future users of Zuul to quickly and easily create jobs of their own. Paul Belanger has been making progress on this in the form of a generic tox job (see https://review.openstack.org/438281), and we're now at the point of discussing why / how this is a foundational example. Additionally, as more of the major subsystems of Zuul complete their refactoring, the ability for one to contribute to Zuul v3 continues to get a bit easier. Even though at the moment we still recommend having deep familiarity with one or more of the subsystems, we do now have some "low-hanging fruit" tasks listed in storyboard. Use this magical link to find your way: https://storyboard.openstack.org/#!/story/list?status=active=low-hanging-fruit_id=679 Some new specs for enhancements and/or features have been put forth as well: * An interface for Zuul Job Reporting https://review.openstack.org/444088 * Zuulv3 Executor Security Enhancement https://review.openstack.org/95 * Update job trees to graphs https://review.openstack.org/443985 (yes, technically this is an *update* to a spec, not a new one.) Upcoming tasks and focus: * Re-enabling disabled tests: We're continuing to make our way through the list of remaining tests that need enabling. See the list, which includes an annotation as to complexity for each test, here: https://etherpad.openstack.org/p/zuulv3skips * Full task list and plan is in the Zuul v3 storyboard: https://storyboard.openstack.org/#!/board/41 Recent changes: * Zuul v3: https://review.openstack.org/#/q/status:closed+project:openstack-infra/zuul+branch:feature/zuulv3,25 (Early adoptors should be aware we renamed zuul-launcher to zuul-executor, this is a breaking change: https://review.openstack.org/#/c/445594) * Nodepool: https://review.openstack.org/#/q/status:closed+project:openstack-infra/nodepool+branch:feature/zuulv3,25 Previous IRC Meeting minutes & logs: * 2017-03-06 Minutes: http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-03-06-22.03.html * 2017-03-06 Full log: http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-03-06-22.03.log.html * 2017-03-13 Minutes: http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-03-13-22.02.html * 2017-03-13 Full log: http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-03-13-22.02.log.html == About Zuul and Zuul v3 == Zuul is a pipeline-oriented project gating system, driving the project automation necessary to enable a continuous integration environment for the OpenStack community. It directs the OpenStack community’s testing, running tens of thousands of jobs each day, responding to events from the code review system and stacking potential changes to be tested together. Testing and continuous integration for drivers is also enabled by OpenStack’s deployment of Zuul for ~50 third-party projects. Zuul v3 is the upcoming, not-yet-in-production version of Zuul currently under development, bringing (amongst many others) the following improvements and features: * simplifying the ability to run jobs in multi-node environments * improved management of large numbers of jobs and job variations * support for in-tree job configuration * ability to define jobs using Ansible (http://github.com/ansible/ansible) Even though prior versions of Zuul are already in use by numerous projects and companies not related to OpenStack efforts, a primary goal of Zuul v3 is to make Zuul a universally useful CI / CD engine for any size use case. The design of Zuul v3 improves the modularity of the system, and enables new triggers (such as GitHub) and reporters (where results are sent) to be more easily integrated with Zuul; additionally, the ability to execute jobs on non-OpenStack clouds, such as, well, the other ones :D, as well as non-cloud
[OpenStack-Infra] Status of Zuul v3
Greetings! Welcome to the first-ever Zuul v3 status update. :) This periodic update is primarily intended as a way to keep contributors to the OpenStack community apprised of Zuul v3 project status, including future changes and milestones on our way to use in production. Additionally, the numerous existing and future users of Zuul outside of the OpenStack community may find this update useful as a way to track Zuul v3 development status. This status update email is anticipated to be sent on an approximately bi-weekly basis, though more frequent updates may occur as we approach in-production dates. == Wait, what’s going on? == Over the coming months, OpenStack’s infrastructure team will be migrating Zuul, OpenStack’s project gating and automation system, from Zuul v2.5 to Zuul v3. There are significant changes associated with this migration, particularly with regards to how jobs are written, as one of those changes is the move from JJB to Ansible. Detailed information about Zuul v3, including specs, motivation, and community processes, is listed below the “project status and updates” portion of this mail. Significant dates, milestones, and changes directly affecting contributors in the OpenStack community will be announced on the appropriate mailing lists for OpenStack (primarily openstack-dev and openstack-infra), as well as in future versions of this project status update. And for those wondering: * Zuul v3 is NOT YET USED IN PRODUCTION. By anyone. OpenStack’s infrastructure team has a test deployment up and running for use in during development (http://zuulv3-dev.openstack.org). OpenStack’s production version of Zuul currently uses Zuul 2.5. * No, you will not have to rewrite all your jobs. We will be automagically converting existing jobs into Ansible jobs, and collaborating directly with project teams in OpenStack in an orderly fashion as we approach the move to in-production status. More on this below. * Yes, this email will be less verbose in the future. :) == Zuul v3 project status and updates == ** It’s alive! ** Last week on Tuesday at the inaugural OpenStack PTG (Feb. 20-24, 2017), for the very first time, Zuul v3 was brought online in a test environment, and a simple hello world job was successfully executed on a single node. That job was immediately followed by a successful “hello worlds” job on multiple nodes. That’s right: Zuul v3 went from single-node to multi-node functionality in a matter of minutes! There was much rejoicing! A super exciting time for the folks who have been working towards this moment. (By the way: if you, fine reader, know anyone who has been hard at work on Zuul v3, send them your congratulations. It’s a Big Deal!) Even though the Infra team’s official days at the PTG were Monday and Tuesday, folks continued hacking on Zuul throughout the rest of the week, getting general tox-based jobs running, and enabling the ability to collect and publish logs. Monty Taylor, known to many of us as mordred, detailed these accomplishments, and outlined the bigger picture of the future of Zuul, in a blog post this week. For the extra-curious, since folks have asked, "What does a job look when we are using Ansible?" -- he has also graciously provided some example code snippets. Read more here: http://inaugust.com/posts/whats-coming-zuulv3.html Also, ** FULL DISCLAIMER ** This may be obvious to some readers, given that this is a progress and status update email, but for all the aspirational, daring folks out there: At this point, Zuul v3 is NOT yet secure or stable. You should not, not, please, DO NOT ATTEMPT TO RUN IT IN PRODUCTION YOURSELF yet. It is still under heavy development, and v3 is not currently used in production for OpenStack. Zuul v3 content (jobs, etc.) should not undergo heavy creation as syntax is still under development and expected to change between now and release date. We know you’re excited. We are too. Just be patient. The time will come. :) Speaking of when that time will come, many folks have asked: When is that time actually coming? Commitment is hard, you see, especially when making it more awesome for people to commit code :) The infra team expects this migration to happen in the next few months. In the meantime: please know that the infrastructure team intends to communicate milestones, information, and significant dates of change to the openstack-dev mailing list, as they would any other significant project infrastructure-related changes. As we move closer to rolling out Zuul v3, significant dates to be aware of, as well as reminders, will be announced broadly, and we will be working with individual project teams to assist them in migration, including the scripted conversion of existing jobs from JJB to Ansible. The vast quantity of documentation in OpenStack that directly or indirectly relates to Zuul, including creating guides for writing future jobs, will also be updated. Upcoming tasks and focus: * Config syntax, including reporting
[openstack-dev] Status of Zuul v3
Greetings! Welcome to the first-ever Zuul v3 status update. :) This periodic update is primarily intended as a way to keep contributors to the OpenStack community apprised of Zuul v3 project status, including future changes and milestones on our way to use in production. Additionally, the numerous existing and future users of Zuul outside of the OpenStack community may find this update useful as a way to track Zuul v3 development status. This status update email is anticipated to be sent on an approximately bi-weekly basis, though more frequent updates may occur as we approach in-production dates. == Wait, what’s going on? == Over the coming months, OpenStack’s infrastructure team will be migrating Zuul, OpenStack’s project gating and automation system, from Zuul v2.5 to Zuul v3. There are significant changes associated with this migration, particularly with regards to how jobs are written, as one of those changes is the move from JJB to Ansible. Detailed information about Zuul v3, including specs, motivation, and community processes, is listed below the “project status and updates” portion of this mail. Significant dates, milestones, and changes directly affecting contributors in the OpenStack community will be announced on the appropriate mailing lists for OpenStack (primarily openstack-dev and openstack-infra), as well as in future versions of this project status update. And for those wondering: * Zuul v3 is NOT YET USED IN PRODUCTION. By anyone. OpenStack’s infrastructure team has a test deployment up and running for use in during development (http://zuulv3-dev.openstack.org). OpenStack’s production version of Zuul currently uses Zuul 2.5. * No, you will not have to rewrite all your jobs. We will be automagically converting existing jobs into Ansible jobs, and collaborating directly with project teams in OpenStack in an orderly fashion as we approach the move to in-production status. More on this below. * Yes, this email will be less verbose in the future. :) == Zuul v3 project status and updates == ** It’s alive! ** Last week on Tuesday at the inaugural OpenStack PTG (Feb. 20-24, 2017), for the very first time, Zuul v3 was brought online in a test environment, and a simple hello world job was successfully executed on a single node. That job was immediately followed by a successful “hello worlds” job on multiple nodes. That’s right: Zuul v3 went from single-node to multi-node functionality in a matter of minutes! There was much rejoicing! A super exciting time for the folks who have been working towards this moment. (By the way: if you, fine reader, know anyone who has been hard at work on Zuul v3, send them your congratulations. It’s a Big Deal!) Even though the Infra team’s official days at the PTG were Monday and Tuesday, folks continued hacking on Zuul throughout the rest of the week, getting general tox-based jobs running, and enabling the ability to collect and publish logs. Monty Taylor, known to many of us as mordred, detailed these accomplishments, and outlined the bigger picture of the future of Zuul, in a blog post this week. For the extra-curious, since folks have asked, "What does a job look when we are using Ansible?" -- he has also graciously provided some example code snippets. Read more here: http://inaugust.com/posts/whats-coming-zuulv3.html Also, ** FULL DISCLAIMER ** This may be obvious to some readers, given that this is a progress and status update email, but for all the aspirational, daring folks out there: At this point, Zuul v3 is NOT yet secure or stable. You should not, not, please, DO NOT ATTEMPT TO RUN IT IN PRODUCTION YOURSELF yet. It is still under heavy development, and v3 is not currently used in production for OpenStack. Zuul v3 content (jobs, etc.) should not undergo heavy creation as syntax is still under development and expected to change between now and release date. We know you’re excited. We are too. Just be patient. The time will come. :) Speaking of when that time will come, many folks have asked: When is that time actually coming? Commitment is hard, you see, especially when making it more awesome for people to commit code :) The infra team expects this migration to happen in the next few months. In the meantime: please know that the infrastructure team intends to communicate milestones, information, and significant dates of change to the openstack-dev mailing list, as they would any other significant project infrastructure-related changes. As we move closer to rolling out Zuul v3, significant dates to be aware of, as well as reminders, will be announced broadly, and we will be working with individual project teams to assist them in migration, including the scripted conversion of existing jobs from JJB to Ansible. The vast quantity of documentation in OpenStack that directly or indirectly relates to Zuul, including creating guides for writing future jobs, will also be updated. Upcoming tasks and focus: * Config syntax, including reporting