Some thoughts:
- Continuous System Integration Test (CSIT) per project, current level is low:
https://jenkins.onap.org/view/CSIT/ We should encourage project as part as
their milestones to have a certain level of coverage through CSIT. Maybe create
a reward, like the project that has the best test coverage (we’ve done this for
a few releases in ODL)
- REST API layer
- E2E functionality within the project itself (using simulators, …)
- CSIT for overall ONAP, e.g run close-loop use cases
- A better way to customize ONAP deployment in OOM. Currently everything is
coming from an onap-parameters.yaml file, which was done initially but isn’t a
good solution. Team is currently working on moving the configuration under each
helm chart (per project), so configuration can be derived from the helm chart
itself. Once done, we can create a common helm chart that will centralize the
ONAP parameters to be passed to any projects helm chart, it will basically sit
on top of the other, as illustrated bellow:
———————
common chart
———————
projects chart
———————
That way, projects chart can be configured to retrieved the value from the
common chart, or to use their own values (using Go Templating, e.g. {{
Values.common.appcversion | default v1.1.1 }} ). So basically we keep the
ability to deploy one chart at the time.
Finally, Helm provides the ability to overwrite the values.yaml properties when
deploying a chart, hence the common values.yaml file can be overwritten to
provide specific environment details.
- Create a CI process for OOM in ONAP’s Jenkins. As OOM is using helm chart for
each application, OOM should produce helm chart, and at release time, should
release helm chart, so consumer (GOCD, Cloudify, …) can consume them. This
currently doesn’t exist in ONAP, moreover, we need ONAP to be able to host helm
chart. Nexus doesn’t offer this capability yet. I know Bintray does, and it’s
on the way for Artifcatory. But we need support from the LF on this to find the
appropriate solution. Note, it could also be a simple HTTP server on which we
can have two folders, snapshot and release, and manipulate helm chart from
there.
- Create a CD process for OOM in ONAP’s Jenkins: Today we have Michael’s script
that does the job. But we should have something that provides more auditing,
better logging, better traceability, etc.... Internally we’ve been using GOCD
for that, and we’re currently looking at how to properly upstream what we’ve
done. GOCD allows to create pipeline, similar to Jenkins pipeline, but geared
mainly for CD and not CI. This will give the ability to deploy ONAP on
multiple/different environments. Also, this could be triggered by an upstream
Jenkins Job, like a merge job, or a keyword comment on a gerrit patch. An
alternative to GOCD, being currently developed in OOM, is Cloudify. So both
options will sit on top of OOM, consuming helm chart build by OOM.
- Ability to upgrade a component. Today, OOM only offers the ability to create
or delete component, which is underusing the advantage of Kubernetes that
support upgrade for any deployment.
With the upgradability feature provided by OOM, we will be able to re-deploy
only components that have changed, based on docker image version. So no need to
re-deploy the whole ONAP. And the best part is, it will rollback is the upgrade
has failed. So we can keep and “stable” like kind of environment where to
periodically run the CSIT covering the close-loop use cases.
Regarding how the process should work in Gerrit:
- I tend to think we shouldn’t have to wait for an hour or so to have a
Verified +1 from Jenkins, this is too long, and this will impact development
cycles badly. Ideally, the Jenkins verified job shouldn’t take more than 30 mn
if we want to be efficient.
The addition of a Gerrit trigger keyword to run it the full CSIT close-loops
suites will allow committer/developers to run more in-depth verification on
patches that they feel requires it.
Alexis
> On Jan 23, 2018, at 2:37 PM, Michael O'Brien <frank.obr...@amdocs.com> wrote:
>
> Team,
> Hi, the Integration(Ran, Marco, Gary, Helen, Brian), OOM(Alexis, Mike O,
> Mike E, Yuri), the Linux Foundation(Jessica) and members of the TSC (Gildas)
> have been discussing build tagging and continuous deployment as a way of
> validating the runtime integrity of each component merge – a fully jenkins
> triggered automated CI/CD system that does more than healthcheck.
> At the last Integration meet and a couple before that we discussed moving
> away from blind tagging and more towards continuous build validation – this
> meeting is in response.
> One of the major goals of this meeting is getting a clean set of tasks we
> will present to the Linux Foundation and the PTLs on any
> addition/modification of the current Jenkins/nexus3 build structure.
>
> We will be holding weekly meetings so everyone can contribute and be
> aware of work items and changes to the way we build, deploy, test and mark
> each commit in onap. Currently the JobBuilder does CI compilation/test
> marking a -1/+1. We will be coming up with a set of proposals to drive CD
> with the goal of creating/consuming a manifest per commit and marking a merge
> as +1/-1 using an additional “CDBuilder”.
> Please come to the meeting to help us align/fix anything out of order
> and sort out the work items.
>
> https://zoom.us/j/7939937123 <https://zoom.us/j/7939937123>
> Tentatively 1130 EST (GMT-5) on Thu – 30 min after the TSC meeting – we can
> move it
>
> There is a JIRA being worked out that describes the tasks needed to
> transition a live POC using what is running now. There is also work being
> done in OOM-460 that will align any CD system to helm based config packaging.
> This meeting is to work out tasks mostly around what will impact the LF and
> PTLs as we bring in continuous deployment and align CI/CD with what is master
> right now.
>
> Agenda items
> Review https://jira.onap.org/browse/OOM-500
> <https://jira.onap.org/browse/OOM-500>
> Discuss aligning the nexus3 docker tag timestamp format
> <https://nexus3.onap.org/> – LF task and PTL oversight
> Discuss building tagged docker images per commit – more granular than daily
> <https://jenkins.onap.org/view/Merge-Jobs/job/aai-sparky-be-master-merge-java/>
> – LF task
> Discuss how to validate a tagset containing the specific commit under test –
> the CD deployment <http://jenkins.onap.info/job/oom-cd> server/job – this is
> where we pull/push the docker tagset manifest under test.
>
> Discussion start (from OOM-500)
>
> <ATT59161 1.jpg> existing flow (actual)
> gerrit review commits - partial-CI runs - JobBuilder marks -1 (compile
> failure, sonar failure)
> gerrit review commits - partial-CI runs - JobBuilder marks +1, committer +2
> merges to master, (later daily docker merge build tagged docker blindly)
> no way to know whether that commit degraded ONAP
> <ATT53870 2.jpg> proposed flow (expected) - two phase JobBuilder +1 process
> gerrit review commits - partial-CI runs - JobBuilder marks -1 (compile
> failure, sonar failure)
> gerrit review commits - partial-CI runs - JobBuilder marks +1, (what we add
> below)
> kick in docker merge immediately, we tag the docker image, we adjust the
> manifest for this review
> run extra CDBuilder that runs CD deploy of OOM using above tagset manifest,
> healthcheck, vFW
> report failure - marks gerrit review as -1 - no tag set published for this
> failed commit
> or report pass - marks gerrit review as +1
> jenkins now retags "latest" to the the docker built above for the review (do
> not blindly tag "latest" to the last docker build whether it passes/fails CD)
> Tagset for that build becomes the latest stable master build of ONAP (as in
> an OOM deploy will run not from master tip but from a tagged set that omits
> the last breaking change)
> committer +2 merges to master, (docker merge and tagging should not be
> required as long as master has not moved during the 1 hour CD build)
>
>
> Thank you
> /michael
>
>
>
>
>
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement,
> you may review at https://www.amdocs.com/about/email-disclaimer
> <https://www.amdocs.com/about/email-disclaimer><Mail
> Attachment.ics>_______________________________________________
> onap-discuss mailing list
> onap-discuss@lists.onap.org <mailto:onap-discuss@lists.onap.org>
> https://lists.onap.org/mailman/listinfo/onap-discuss
> <https://lists.onap.org/mailman/listinfo/onap-discuss>
_______________________________________________
onap-discuss mailing list
onap-discuss@lists.onap.org
https://lists.onap.org/mailman/listinfo/onap-discuss
