Thanks for the feedback and ideas of solving our challenges with running OPNFV as a VNF development and evaluation platform.
I was not trying to prescribe a specific solution, but wanted to call out the challenges that we are encountering when running OPFNV in our NFV labs. The direction of Tim and Frank’s feedback could help address these challenges. Enabling the installer teams to focus on maintenance post release. Enabling scenario teams to rollout new scenarios in maintenance releases that do NOT negatively impact the stability of the currently supported scenarios could be a reasonable compromise. Or just following the strict model as outlined by Ulrich would help with the stability. Please don’t break backwards compatibility of the configuration files during maintenance releases. Changing configuration files with major releases would be manageable. Many Thanks, Randy Levensalor | [email protected] CableLabs® | o 303.661.3455 | c 970.214.1316 On 2/22/17, 8:18 AM, "Frank Brockners (fbrockne)" <[email protected]> wrote: That makes sense. The whole point of micro-releases would be to release scenarios once they are ready - and it falls into the responsibility of a scenario owner to shepherd the process. Micro-releases should *not* be a vehicle to push other teams to go out of their way. Frank -----Original Message----- From: Tim Rozet [mailto:[email protected]] Sent: Mittwoch, 22. Februar 2017 15:53 To: Ulrich Kleber <[email protected]> Cc: Randy Levensalor <[email protected]>; Frank Brockners (fbrockne) <[email protected]>; David McBride <[email protected]>; TECH-DISCUSS OPNFV <[email protected]>; [email protected] Subject: Re: [opnfv-tech-discuss] [release] E-release schedule I think having maintenance releases makes sense. From an installer perspective, if folks want to add a new scenario for a maintenance release then they need to carry most of the weight of integrating it. It is hard for a small team like Apex to juggle adding features for a previous release while rushing to get the next version of OS to work for the following release. Bug fixing and documentation updates I think we are fine with for maintenance releases. Tim Rozet Red Hat SDN Team ----- Original Message ----- From: "Ulrich Kleber" <[email protected]> To: "Randy Levensalor" <[email protected]>, "Frank Brockners (fbrockne)" <[email protected]>, "David McBride" <[email protected]>, "TECH-DISCUSS OPNFV" <[email protected]>, [email protected] Sent: Wednesday, February 22, 2017 8:00:09 AM Subject: Re: [opnfv-tech-discuss] [release] E-release schedule Hi, what Randy explains would mean that we should move to a very strict concept of maintenance releases. That would mean we carry through our milestones properly to get the 1.0 release out. After that we should only do bug fixing. Not a single feature. If we do that, 2.0 and 3.0 will not take big efforts, but each provide better stability. It shouldn’t even be a problem to do 4.0 if there are urgent fixes. But we should be strict then: Every feature that misses the 1.0, has to wait for the next major release. Just my 2 cents. Cheers, Uli From: [email protected] [mailto:[email protected]] On Behalf Of Randy Levensalor Sent: Friday, 17 February, 2017 00:16 To: Frank Brockners (fbrockne); David McBride; TECH-DISCUSS OPNFV; [email protected] Subject: Re: [opnfv-tech-discuss] [release] E-release schedule Frank, +1 on your suggestions. David, I applaud your effort to reduce the overhead for the community. TL;DR Today I spend > 80% of my time getting OPNFV to run and I’d prefer to spend > 80% of my time running VNFs on OPNFV. As a frequent user, use case 1 that Frank defined aligns with my primary goal for OPNFV. “(1) User/consumer of a readily integrated NFV stack.” The need for a working platform currently outweigh the need for additional features. I suspect that as the platform and our use cases mature, some of the incremental features will become a higher priority. With the one release every 6 months, I fear that I would have to carry a large patch set or something to have a functioning platform. To add a little context to my reservations about the reduced number of releases, I’d like to share my experience below as an example of how the release cycle can impact users. Please don’t read this as a complaint or detracting from the project’s progress and achievement. The fact that we can run a 100% open source VIM and NFVI with a very limited staff is a minor miracle. There may some other clever solutions. Fatih and Jack have mentions some options that could help. With the Arno, Brahmaputra and Colorado releases we would start downloading release candidates and kick the tires. We try and look at multiple installers, but most of our time has been spent on Apex. The RC installs almost never work. About 1/3 rd of these failures are due to hardware or user error that are not easy to debug. For instance, a bad NIC on a compute node can cause the controller install to fail. The rest are defects or documentation issues. The1.0 release goes better. I can usually get a few patches into the first release to resolve critical issue and the community is working day and night to get the release out the door. By the 2.0 release, most of the defects are fixed. There are inevitably a few issues that take more than a month to root cause. Some of these more challenging defects, which are often related to platform stability, cannot be fixed until the next release. Resources are already focused on adding new features and do not have the time nor desire to backport these fixes. The longest that I have kept an OPNFV Colorado instance running was 32 days. That required the Colorado 3.0 release, manually applying a few patches that will be in Danube and limiting the type activities. Now we are preparing to kick the tires on Danube continue with difficulties keeping our lab running long enough to deploy an interesting use case. In the meantime, we will continue to reinstall at least once a month and look forward to the Danube release. Randy Levensalor | [email protected] Cable Labs® | o 303.661.3455 | c 970.214.1316 From: < [email protected] > on behalf of "Frank Brockners (fbrockne)" < [email protected] > Date: Thursday, February 16, 2017 at 1:36 AM To: David McBride < [email protected] >, OPNFV Tech Discussion < [email protected] >, " [email protected] " < [email protected] > Subject: Re: [opnfv-tech-discuss] [release] E-release schedule David, thanks for the summary. Let’s remind ourselves, that in OPNFV we’re really trying to meet the needs of two different audiences: (1) User/consumer of a readily integrated NFV stack – as well as marketing operations (2) Developer/tester of an NFV stack. Audience #1 is mostly interested in stability, even if that means that things are released a little later (i.e. you build on long released components). Audience #2 is pushing the envelope and requires the ability to evolve/develop and integrate the latest set of components; once working they desire to release things to allow others to build on top; and move on/start over. The current 1.0/2.0/3.0 was an effort to meet the needs of both audiences, i.e. · Have a “major” release. · Allow developers to release scenarios when they are ready and evolve, without too much of a maintenance burden. This is also why we typically did not fix component versions for a release, but said: Based or ODL Boron or later. I agree that releases are not free – especially the “major” release, because it comes with significant documentation and coordination needs. That said, it is mostly the “major” release with a lot of central coordination which creates efforts. Labeling and pushing an updated version of test results and documentation is relatively low effort – and can even be done by a scenario team. It does not even require central coordination. And our pipeline is now mature enough to do these things with low/moderate overhead. So rather than move back in history and go back to a single release every 6 months, which will make OPNFV a very inflexible organization for developers, I would strongly suggest that we rather consider evolving the current release process. IMHO we should be ready to have monthly micro-releases (scenario owners publish those scenarios which are “ready”, i.e. have docs ready and pass testing), and every 6 months we do a macro-release (with marketing/press announcement) which includes the set of scenarios which are “ready” by then. Macro-releases can be coupled to certain upstream component versions (as selection criteria for what is in/out of a macro release) – whereas micro-release would enjoy complete freedom. Thoughts? Thanks, Frank From: David McBride [ mailto:[email protected] ] Sent: Mittwoch, 15. Februar 2017 20:26 To: TECH-DISCUSS OPNFV < [email protected] >; [email protected] Cc: Frank Brockners (fbrockne) < [email protected] >; Tapio Tallgren < [email protected] > Subject: [release] E-release schedule Greetings, During the TSC call, yesterday, I took an action to start an email discussion about the schedule for the E-release . Specifically, I suggested that we just plan for a single release, rather than three releases, as we've done in the past. Then, when the release date approaches, we evaluate whether we need a point release, then schedule it at that time. Why? * Scheduling three releases has created a lot of confusion with the project teams The purpose of the three releases is to give project teams time to debug and fix scenarios that are not ready for 1.0. They are not separate development timelines with separate release milestones. However, many believe that it isn't necessary to meet release milestones, because they will simply shift to the 2.0 or 3.0 release. * In the past two releases, the new content released in 2.0 has been minimal. For example, for Colorado 2.0, just two new scenarios were released. Human nature is such that, given the opportunity for a later deadline, many will take it. * Releases are not free. In addition to the overhead required for labeling, creating ISOs, and updating documentation, projects that released in previous releases, are required to update their code for subsequent releases to resolve any issues, even if they weren't intending to do any additional work on that major release. For example, let's say that a project releases in Danube 1.0, they're satisfied with their effort, so they shift their focus to the E-release. However, changes after 1.0 break their scenario. So, suddenly, they find themselves working on Danube 2.0, even though they aren't releasing any new scenarios. This process repeats for Danube 3.0. During the TSC call, it was suggested that a 2.0 or 3.0 release provides an opportunity to integrate a late release of a major upstream component (e.g. ODL). However, this is counter to our previous agreement not to change major upstream components after the 1.0 release. Unfortunately, this happened in Colorado and created significant disruption, including a slip in the 2.0 release. Per our discussion on Tuesday, I've created a wiki page to capture pros and cons of various schedule options. Feel free to edit it and add your thoughts. David -- David McBride Release Manager, OPNFV Mobile: +1.805.276.8018 Email/Google Talk: [email protected] Skype: davidjmcbride1 IRC: dmcbride _______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss _______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
