Hello community, here is the log from the commit of package helmfile for openSUSE:Factory checked in at 2020-08-18 12:00:10 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/helmfile (Old) and /work/SRC/openSUSE:Factory/.helmfile.new.3399 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "helmfile" Tue Aug 18 12:00:10 2020 rev:4 rq:827382 version:0.125.7 Changes: -------- --- /work/SRC/openSUSE:Factory/helmfile/helmfile.changes 2020-08-06 10:43:34.134163574 +0200 +++ /work/SRC/openSUSE:Factory/.helmfile.new.3399/helmfile.changes 2020-08-18 12:02:35.231419735 +0200 @@ -1,0 +2,30 @@ +Mon Aug 17 15:38:34 UTC 2020 - Manfred Hollstein <[email protected]> + +- v0.125.7: Fix regression installing raw K8s manifests (#1417) + * a9aa7af (HEAD, tag: v0.125.7, origin/master, origin/HEAD, master) + Fix regression installing raw K8s manifests (#1417) + * 738409e Fix regression for helmfile template with helm v2 since + v0.125.4 (#1416) + +- v0.125.6: Revert filtering on repositories to be updated (#1412) + * ec87a39 (HEAD, tag: v0.125.6, origin/master, origin/HEAD, master) + Revert filtering on repositories to be updated (#1412) + * ab1c118 Bump chartify to v0.4.0 (#1409) + * c8c444d Add test case (#1410) + +- v0.125.5: Fix chart fetched by go-getter not to fail due to missing + dependencies + * 9a03d79 (HEAD, tag: v0.125.5, origin/master, origin/HEAD, master) + Fix chart fetched by go-getter not to fail due to missing + dependencies (#1408) + +- v0.125.4: Fix race on/sometimes missing postsync and cleanup hooks + (#1407) + * 0ef7e65 (HEAD, tag: v0.125.4, origin/master, origin/HEAD, master) + Fix race on/sometimes missing postsync and cleanup hooks (#1407) + * 61b61d3 Fix helmfile template with Helm 3 ignoring chart version + (#1406) + * 2710cb3 Fix go-getter URL in chart to actually work (#1405) + * 4d7fcd8 doc: ArgoCD integration (#1402) + +------------------------------------------------------------------- Old: ---- helmfile-0.125.3.tar.gz New: ---- helmfile-0.125.7.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ helmfile.spec ++++++ --- /var/tmp/diff_new_pack.R7YUt2/_old 2020-08-18 12:02:36.623419925 +0200 +++ /var/tmp/diff_new_pack.R7YUt2/_new 2020-08-18 12:02:36.623419925 +0200 @@ -16,9 +16,9 @@ # Please submit bugfixes or comments via https://bugs.opensuse.org/ # -%define git_commit b85243a6b4a117ca253a877bcd7eb4e0f892f3ea +%define git_commit a9aa7af5725ddf95b73b783202ec953897d67240 Name: helmfile -Version: 0.125.3 +Version: 0.125.7 Release: 0 Summary: Deploy Kubernetes Helm Charts License: MIT ++++++ _service ++++++ --- /var/tmp/diff_new_pack.R7YUt2/_old 2020-08-18 12:02:36.659419930 +0200 +++ /var/tmp/diff_new_pack.R7YUt2/_new 2020-08-18 12:02:36.659419930 +0200 @@ -5,7 +5,7 @@ <param name="exclude">.git</param> <param name="versionformat">@PARENT_TAG@</param> <param name="versionrewrite-pattern">v(.*)</param> - <param name="revision">v0.125.3</param> + <param name="revision">v0.125.7</param> <param name="changesgenerate">enable</param> </service> <service name="recompress" mode="disabled"> ++++++ _servicedata ++++++ --- /var/tmp/diff_new_pack.R7YUt2/_old 2020-08-18 12:02:36.695419935 +0200 +++ /var/tmp/diff_new_pack.R7YUt2/_new 2020-08-18 12:02:36.695419935 +0200 @@ -1,4 +1,4 @@ <servicedata> <service name="tar_scm"> <param name="url">https://github.com/roboll/helmfile.git</param> - <param name="changesrevision">b85243a6b4a117ca253a877bcd7eb4e0f892f3ea</param></service></servicedata> + <param name="changesrevision">a9aa7af5725ddf95b73b783202ec953897d67240</param></service></servicedata> ++++++ helmfile-0.125.3.tar.gz -> helmfile-0.125.7.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/.circleci/config.yml new/helmfile-0.125.7/.circleci/config.yml --- old/helmfile-0.125.3/.circleci/config.yml 2020-08-06 02:06:25.000000000 +0200 +++ new/helmfile-0.125.7/.circleci/config.yml 2020-08-14 12:52:00.000000000 +0200 @@ -153,7 +153,7 @@ - run: name: Install helm environment: - HELM_VERSION: v3.0.0 + HELM_VERSION: v3.1.0 command: | HELM_FILENAME="helm-${HELM_VERSION}-linux-amd64.tar.gz" curl -Lo ${HELM_FILENAME} "https://get.helm.sh/${HELM_FILENAME}"; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/README.md new/helmfile-0.125.7/README.md --- old/helmfile-0.125.3/README.md 2020-08-06 02:06:25.000000000 +0200 +++ new/helmfile-0.125.7/README.md 2020-08-14 12:52:00.000000000 +0200 @@ -963,19 +963,21 @@ - `cleanup` Hooks associated to `prepare` events are triggered after each release in your helmfile is loaded from YAML, before execution. - -Hooks associated to `cleanup` events are triggered after each release is processed. +`prepare` hooks are triggered on the release as long as it is not excluded by the helmfile selector(e.g. `helmfile -l key=value`). Hooks associated to `presync` events are triggered before each release is applied to the remote cluster. This is the ideal event to execute any commands that may mutate the cluster state as it will not be run for read-only operations like `lint`, `diff` or `template`. `preuninstall` hooks are triggered immediately before a release is uninstalled as part of `helmfile apply`, `helmfile sync`, `helmfile delete`, and `helmfile destroy`. -Hooks associated to `postsync` events are triggered after each release is applied to the remote cluster. -This is the ideal event to execute any commands that may mutate the cluster state as it will not be run for read-only operations like `lint`, `diff` or `template`. - `postuninstall` hooks are triggered immediately after successful uninstall of a release while running `helmfile apply`, `helmfile sync`, `helmfile delete`, `helmfile destroy`. +`postsync` hooks are triggered after each release is synced(installed, updated, or uninstalled) to/from the cluster, regardless of the sync was successful or not. +This is the ideal place to execute any commands that may mutate the cluster state as it will not be run for read-only operations like `lint`, `diff` or `template`. + +`cleanup` hooks are triggered after each release is processed. +This is the counterpart to `prepare`, as any release on which `prepare` has been triggered gets `cleanup` triggered as well. + The following is an example hook that just prints the contextual information provided to hook: ``` @@ -1142,6 +1144,36 @@ - [renovate](https://github.com/renovatebot/renovate) automates chart version updates. See [this PR for more information](https://github.com/renovatebot/renovate/pull/5257). - For updating container image tags and git tags embedded within helmfile.yaml and values, you can use [renovate's regexManager](https://docs.renovatebot.com/modules/manager/regex/). Please see [this comment in the renovate repository](https://github.com/renovatebot/renovate/issues/6130#issuecomment-624061289) for more information. +- [ArgoCD Integration](#argocd-integration) + +### ArgoCD Integration + +Use [ArgoCD](https://argoproj.github.io/argo-cd/) with `helmfile template` for GitOps. + +ArgoCD has support for kustomize/manifests/helm chart by itself. Why bother with Helmfile? + +The reasons may vary: + +1. You do want to manage applications with ArgoCD, while letting Helmfile manage infrastructure-related components like Calico/Cilium/WeaveNet, Linkerd/Istio, and ArgoCD itself. + - This way, any application deployed by ArgoCD has access to all the infrastructure. + - Of course, you can use ArgoCD's [Sync Waves and Phases](https://argoproj.github.io/argo-cd/user-guide/sync-waves/) for ordering the infrastructure and application installations. But it may be difficult to separate the concern between the infrastructure and apps and annotate K8s resources consistently when you have different teams for managing infra and apps. +2. You want to review the exact K8s manifests being applied on pull-request time, before ArgoCD syncs. + - This is often better than using a kind of `HelmRelease` custom resources that obfuscates exactly what manifests are being applied, which makes reviewing harder. +3. Use Helmfile as the single-pane of glass for all the K8s resources deployed to your cluster(s). + - Helmfile can reduce repetition in K8s manifests across ArgoCD application + +For 1, you run `helmfile apply` on CI to deploy ArgoCD and the infrastructure components. + +> helmfile config for this phase often reside within the same directory as your Terraform project. So connecting the two with [terraform-provider-helmfile](https://github.com/mumoshu/terraform-provider-helmfile) may be helpful + +For 2, another app-centric CI or bot should run `helmfile template --output-dir-template gitops//{{.Release.Name}} && cd gitops && git add . && git commit && git push` to render/commit manifests, +so that they can be deployed by Argo CD as usual. + +Recommendations: + +- Do create ArgoCD `Application` custom resource per Helm/Helmfile release, each point to respective sub-directory generated by `helmfile template --output-dir-template` +- If you don't directly push it to the main Git branch and instead go through a pull-request, do lint rendered manifests on your CI, so that you can catch easy mistakes earlier/before ArgoCD finally deploys it +- See [this ArgoCD issue](https://github.com/argoproj/argo-cd/issues/2143#issuecomment-570478329) for why you may want this, and see [this helmfile issue](https://github.com/roboll/helmfile/pull/1357) for how `--output-dir-template` works. # Attribution diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/go.mod new/helmfile-0.125.7/go.mod --- old/helmfile-0.125.3/go.mod 2020-08-06 02:06:25.000000000 +0200 +++ new/helmfile-0.125.7/go.mod 2020-08-14 12:52:00.000000000 +0200 @@ -24,7 +24,7 @@ github.com/spf13/cobra v0.0.3 github.com/tatsushid/go-prettytable v0.0.0-20141013043238-ed2d14c29939 github.com/urfave/cli v1.20.0 - github.com/variantdev/chartify v0.3.11 + github.com/variantdev/chartify v0.4.1 github.com/variantdev/dag v0.0.0-20191028002400-bb0b3c785363 github.com/variantdev/vals v0.10.0 go.uber.org/multierr v1.1.0 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/go.sum new/helmfile-0.125.7/go.sum --- old/helmfile-0.125.3/go.sum 2020-08-06 02:06:25.000000000 +0200 +++ new/helmfile-0.125.7/go.sum 2020-08-14 12:52:00.000000000 +0200 @@ -519,6 +519,7 @@ github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= github.com/imdario/mergo v0.3.9 h1:UauaLniWCFHWd+Jp9oCEkTBj8VO/9DKg3PV3VCNMDIg= github.com/imdario/mergo v0.3.9/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= +github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM= github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= github.com/influxdata/influxdb v0.0.0-20190411212539-d24b7ba8c4c4 h1:3K3KcD4S6/Y2hevi70EzUTNKOS3cryQyhUnkjE6Tz0w= github.com/influxdata/influxdb v0.0.0-20190411212539-d24b7ba8c4c4/go.mod h1:qZna6X/4elxqT3yI9iZYdZrWWdeFOOprn86kgg4+IzY= @@ -777,6 +778,7 @@ github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/stretchr/testify v1.5.0 h1:DMOzIV76tmoDNE9pX6RSN0aDtCYeCg5VueieJaAo1uw= github.com/stretchr/testify v1.5.0/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= +github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4= github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA= github.com/tatsushid/go-prettytable v0.0.0-20141013043238-ed2d14c29939 h1:BhIUXV2ySTLrKgh/Hnts+QTQlIbWtomXt3LMdzME0A0= @@ -827,6 +829,10 @@ github.com/variantdev/chartify v0.3.10/go.mod h1:0tw+4doFHsNnhttYx7I9Pv/dsZ82BD4UuTV9saBOcfw= github.com/variantdev/chartify v0.3.11 h1:+jknp9CQ4kvgVMifuwxuu8/5UAVajuNrPBLWnTCTNpA= github.com/variantdev/chartify v0.3.11/go.mod h1:0tw+4doFHsNnhttYx7I9Pv/dsZ82BD4UuTV9saBOcfw= +github.com/variantdev/chartify v0.4.0 h1:Ze+gfykTqFuQgfH2w6/ZqMwsrGrRU8yIsW8M+e7jAHg= +github.com/variantdev/chartify v0.4.0/go.mod h1:Fr4oPNJ4b19knlovn0wBoU2+MZxQdBacmBs30wwgAFk= +github.com/variantdev/chartify v0.4.1 h1:0a8Ipf7n65mbQaAAfI9JEbdDauMnwKs0Gdq/yu5jI7I= +github.com/variantdev/chartify v0.4.1/go.mod h1:Fr4oPNJ4b19knlovn0wBoU2+MZxQdBacmBs30wwgAFk= github.com/variantdev/dag v0.0.0-20191028002400-bb0b3c785363 h1:KrfQBEUn+wEOQ/6UIfoqRDvn+Q/wZridQ7t0G1vQqKE= github.com/variantdev/dag v0.0.0-20191028002400-bb0b3c785363/go.mod h1:pH1TQsNSLj2uxMo9NNl9zdGy01Wtn+/2MT96BrKmVyE= github.com/variantdev/vals v0.4.0 h1:O1O7/sWhlvozcY2DjZBzlE1notxwVo6UBT1+w7HsO/k= @@ -1077,6 +1083,7 @@ google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= google.golang.org/api v0.21.0 h1:zS+Q/CJJnVlXpXQVIz+lH0ZT2lBuT2ac7XD8Y/3w6hY= google.golang.org/api v0.26.0 h1:VJZ8h6E8ip82FRpQl848c5vAadxlTXrUh8RzQzSRm08= +google.golang.org/api v0.30.0 h1:yfrXXP61wVuLb0vBcG6qaOoIoqYEzOQS8jum51jkv2w= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.2.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.3.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/pkg/app/app.go new/helmfile-0.125.7/pkg/app/app.go --- old/helmfile-0.125.3/pkg/app/app.go 2020-08-06 02:06:25.000000000 +0200 +++ new/helmfile-0.125.7/pkg/app/app.go 2020-08-14 12:52:00.000000000 +0200 @@ -987,6 +987,23 @@ return false, false, errs } + releasesWithNoChange := map[string]state.ReleaseSpec{} + for _, r := range toApply { + id := state.ReleaseToID(&r) + _, uninstalled := releasesToBeUpdated[id] + _, updated := releasesToBeDeleted[id] + if !uninstalled && !updated { + releasesWithNoChange[id] = r + } + } + + for id := range releasesWithNoChange { + r := releasesWithNoChange[id] + if _, err := st.TriggerCleanupEvent(&r, "apply"); err != nil { + a.Logger.Warnf("warn: %v\n", err) + } + } + if releasesToBeDeleted == nil && releasesToBeUpdated == nil { if infoMsg != nil { logger := c.Logger() @@ -1091,6 +1108,22 @@ releasesToDelete[id] = r } + releasesWithNoChange := map[string]state.ReleaseSpec{} + for _, r := range toSync { + id := state.ReleaseToID(&r) + _, uninstalled := releasesToDelete[id] + if !uninstalled { + releasesWithNoChange[id] = r + } + } + + for id := range releasesWithNoChange { + r := releasesWithNoChange[id] + if _, err := st.TriggerCleanupEvent(&r, "delete"); err != nil { + a.Logger.Warnf("warn: %v\n", err) + } + } + names := make([]string, len(toSync)) for i, r := range toSync { names[i] = fmt.Sprintf(" %s (%s)", r.Name, r.Chart) @@ -1175,6 +1208,23 @@ releasesToUpdate[id] = r } + releasesWithNoChange := map[string]state.ReleaseSpec{} + for _, r := range toSync { + id := state.ReleaseToID(&r) + _, uninstalled := releasesToDelete[id] + _, updated := releasesToUpdate[id] + if !uninstalled && !updated { + releasesWithNoChange[id] = r + } + } + + for id := range releasesWithNoChange { + r := releasesWithNoChange[id] + if _, err := st.TriggerCleanupEvent(&r, "sync"); err != nil { + a.Logger.Warnf("warn: %v\n", err) + } + } + names := []string{} for _, r := range releasesToUpdate { names = append(names, fmt.Sprintf(" %s (%s) UPDATED", r.Name, r.Chart)) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/pkg/app/app_test.go new/helmfile-0.125.7/pkg/app/app_test.go --- old/helmfile-0.125.3/pkg/app/app_test.go 2020-08-06 02:06:25.000000000 +0200 +++ new/helmfile-0.125.7/pkg/app/app_test.go 2020-08-14 12:52:00.000000000 +0200 @@ -2490,6 +2490,7 @@ var wantRepos = []mockRepo{ {Name: "stable"}, + {Name: "stable2"}, } var buffer bytes.Buffer diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/pkg/remote/remote_test.go new/helmfile-0.125.7/pkg/remote/remote_test.go --- old/helmfile-0.125.3/pkg/remote/remote_test.go 2020-08-06 02:06:25.000000000 +0200 +++ new/helmfile-0.125.7/pkg/remote/remote_test.go 2020-08-14 12:52:00.000000000 +0200 @@ -2,6 +2,7 @@ import ( "fmt" + "github.com/google/go-cmp/cmp" "github.com/roboll/helmfile/pkg/helmexec" "github.com/roboll/helmfile/pkg/testhelper" "os" @@ -154,6 +155,75 @@ } }) } +} + +func TestParse(t *testing.T) { + type testcase struct { + input string + getter, scheme, dir, file, query string + err string + } + + testcases := []testcase{ + { + input: "raw/incubator", + err: "parse url: missing scheme - probably this is a local file path? raw/incubator", + }, + { + input: "git::https://github.com/stakater/Forecastle.git@deployments/kubernetes/chart/forecastle?ref=v1.0.54";, + getter: "git", + scheme: "https", + dir: "/stakater/Forecastle.git", + file: "deployments/kubernetes/chart/forecastle", + query: "ref=v1.0.54", + }, + } + + for i := range testcases { + tc := testcases[i] + + t.Run(fmt.Sprintf("case %d", i), func(t *testing.T) { + src, err := Parse(tc.input) + + var errMsg string + if err != nil { + errMsg = err.Error() + } + + if diff := cmp.Diff(tc.err, errMsg); diff != "" { + t.Fatalf("Unexpected error:\n%s", diff) + } + + var getter, scheme, dir, file, query string + if src != nil { + getter = src.Getter + scheme = src.Scheme + dir = src.Dir + file = src.File + query = src.RawQuery + } + + if diff := cmp.Diff(tc.getter, getter); diff != "" { + t.Fatalf("Unexpected getter:\n%s", diff) + } + + if diff := cmp.Diff(tc.scheme, scheme); diff != "" { + t.Fatalf("Unexpected scheme:\n%s", diff) + } + + if diff := cmp.Diff(tc.file, file); diff != "" { + t.Fatalf("Unexpected file:\n%s", diff) + } + + if diff := cmp.Diff(tc.dir, dir); diff != "" { + t.Fatalf("Unexpected dir:\n%s", diff) + } + + if diff := cmp.Diff(tc.query, query); diff != "" { + t.Fatalf("Unexpected query:\n%s", diff) + } + }) + } } type testGetter struct { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/pkg/state/create_test.go new/helmfile-0.125.7/pkg/state/create_test.go --- old/helmfile-0.125.3/pkg/state/create_test.go 2020-08-06 02:06:25.000000000 +0200 +++ new/helmfile-0.125.7/pkg/state/create_test.go 2020-08-14 12:52:00.000000000 +0200 @@ -250,6 +250,8 @@ []bool{false, true, false}}, {LabelFilter{negativeLabels: [][]string{[]string{"stage", "pre"}, []string{"stage", "post"}}}, []bool{false, false, true}}, + {LabelFilter{negativeLabels: [][]string{[]string{"foo", "bar"}}}, + []bool{false, true, true}}, } state, err := createFromYaml(yamlContent, yamlFile, DefaultEnv, logger) if err != nil { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/pkg/state/helmx.go new/helmfile-0.125.7/pkg/state/helmx.go --- old/helmfile-0.125.3/pkg/state/helmx.go 2020-08-06 02:06:25.000000000 +0200 +++ new/helmfile-0.125.7/pkg/state/helmx.go 2020-08-14 12:52:00.000000000 +0200 @@ -36,52 +36,50 @@ type Chartify struct { Opts *chartify.ChartifyOpts - Chart string Clean func() } -func (st *HelmState) PrepareChartify(helm helmexec.Interface, release *ReleaseSpec, workerIndex int) (*Chartify, func(), error) { - chartify := &Chartify{ - Opts: &chartify.ChartifyOpts{ - WorkaroundOutputDirIssue: true, - EnableKustomizeAlphaPlugins: true, - ChartVersion: release.Version, - Namespace: release.Namespace, - }, - } - - var filesNeedCleaning []string - - clean := func() { - st.removeFiles(filesNeedCleaning) - } - - var shouldRun bool - - chart := release.Chart - if release.Directory != "" && chart == "" { - chart = release.Directory +func (st *HelmState) goGetterChart(chart, dir string, force bool) (string, error) { + if dir != "" && chart == "" { + chart = dir } _, err := remote.Parse(chart) if err != nil { - if release.ForceGoGetter { - return nil, clean, fmt.Errorf("Parsing url from directory of release %q failed due to error %q.\nContinuing the process assuming this is a regular Helm chart or a local directory.", release.Name, err.Error()) + if force { + return "", fmt.Errorf("Parsing url from dir failed due to error %q.\nContinuing the process assuming this is a regular Helm chart or a local dir.", err.Error()) } } else { r := remote.NewRemote(st.logger, st.basePath, st.readFile, directoryExistsAt, fileExistsAt) fetchedDir, err := r.Fetch(chart) if err != nil { - return nil, clean, fmt.Errorf("fetching %q: %v", chart, err) + return "", fmt.Errorf("fetching %q: %v", chart, err) } chart = fetchedDir + } + + return chart, nil +} - filesNeedCleaning = append(filesNeedCleaning, fetchedDir) +func (st *HelmState) PrepareChartify(helm helmexec.Interface, release *ReleaseSpec, chart string, workerIndex int) (*Chartify, func(), error) { + chartify := &Chartify{ + Opts: &chartify.ChartifyOpts{ + WorkaroundOutputDirIssue: true, + EnableKustomizeAlphaPlugins: true, + ChartVersion: release.Version, + Namespace: release.Namespace, + }, + } + + var filesNeedCleaning []string + + clean := func() { + st.removeFiles(filesNeedCleaning) } - chartify.Chart = chart + var shouldRun bool dir := filepath.Join(st.basePath, chart) if stat, _ := os.Stat(dir); stat != nil && stat.IsDir() { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/pkg/state/state.go new/helmfile-0.125.7/pkg/state/state.go --- old/helmfile-0.125.3/pkg/state/state.go 2020-08-06 02:06:25.000000000 +0200 +++ new/helmfile-0.125.7/pkg/state/state.go 2020-08-14 12:52:00.000000000 +0200 @@ -330,15 +330,10 @@ } func (st *HelmState) SyncRepos(helm RepoUpdater, shouldSkip map[string]bool) ([]string, error) { - shouldUpdate, err := st.getRepositoriesToSync() - if err != nil { - return nil, err - } - var updated []string for _, repo := range st.Repositories { - if !shouldUpdate[repo.Name] || shouldSkip[repo.Name] { + if shouldSkip[repo.Name] { continue } @@ -596,19 +591,19 @@ m.Unlock() } - if relErr == nil { - results <- syncResult{} - } else { - results <- syncResult{errors: []*ReleaseError{relErr}} - } - if _, err := st.triggerPostsyncEvent(release, relErr, "sync"); err != nil { st.logger.Warnf("warn: %v\n", err) } - if _, err := st.triggerCleanupEvent(release, "sync"); err != nil { + if _, err := st.TriggerCleanupEvent(release, "sync"); err != nil { st.logger.Warnf("warn: %v\n", err) } + + if relErr == nil { + results <- syncResult{} + } else { + results <- syncResult{errors: []*ReleaseError{relErr}} + } } }, func() { @@ -722,19 +717,19 @@ } } - if relErr == nil { - results <- syncResult{} - } else { - results <- syncResult{errors: []*ReleaseError{relErr}} - } - if _, err := st.triggerPostsyncEvent(release, relErr, "sync"); err != nil { st.logger.Warnf("warn: %v\n", err) } - if _, err := st.triggerCleanupEvent(release, "sync"); err != nil { + if _, err := st.TriggerCleanupEvent(release, "sync"); err != nil { st.logger.Warnf("warn: %v\n", err) } + + if relErr == nil { + results <- syncResult{} + } else { + results <- syncResult{errors: []*ReleaseError{relErr}} + } } }, func() { @@ -819,6 +814,7 @@ releaseName string chartPath string err error + diagnostic string } errs := []error{} @@ -838,6 +834,8 @@ } *st = *updated + var diagnostics []string + st.scatterGather( concurrency, len(releases), @@ -859,22 +857,32 @@ return } - var chartPath string + chartName := release.Chart + + chartPath, err := st.goGetterChart(chartName, release.Directory, release.ForceGoGetter) + if err != nil { + results <- &downloadResults{err: fmt.Errorf("release %q: %w", release.Name, err)} + return + } + + chartFetchedByGoGetter := chartPath != chartName - chartification, clean, err := st.PrepareChartify(helm, release, workerIndex) + chartification, clean, err := st.PrepareChartify(helm, release, chartPath, workerIndex) defer clean() if err != nil { results <- &downloadResults{err: err} return } + var diagnostic string + if chartification != nil { c := chartify.New( chartify.HelmBin(st.DefaultHelmBinary), chartify.UseHelm3(helm3), ) - out, err := c.Chartify(release.Name, chartification.Chart, chartify.WithChartifyOpts(chartification.Opts)) + out, err := c.Chartify(release.Name, chartPath, chartify.WithChartifyOpts(chartification.Opts)) if err != nil { results <- &downloadResults{err: err} return @@ -882,20 +890,63 @@ // TODO Chartify chartPath = out } - } else if !forceDownload { - chartPath = release.Chart - } else if pathExists(normalizeChart(st.basePath, release.Chart)) { - chartPath = normalizeChart(st.basePath, release.Chart) + } else if pathExists(normalizeChart(st.basePath, chartPath)) { + // At this point, we are sure that chartPath is a local directory containing either: + // - A remote chart fetched by go-getter or + // - A local chart + // + // The chart may have Chart.yaml(and requirements.yaml for Helm 2), and optionally Chart.lock/requirements.lock, + // but no `charts/` directory populated at all, or a subet of chart dependencies are missing in the directory. + // + // In such situation, Helm fails with an error like: + // Error: found in Chart.yaml, but missing in charts/ directory: cert-manager, prometheus, postgresql, gitlab-runner, grafana, redis + // + // (See also https://github.com/roboll/helmfile/issues/1401#issuecomment-670854495) + // + // To avoid it, we need to call a `helm dep build` command on the chart. + // But the command may consistently fail when an outdated Chart.lock exists. + // + // (I've mentioned about such case in https://github.com/roboll/helmfile/pull/1400.) + // + // Trying to run `helm dep build` on the chart regardless of if it's from local or remote is + // problematic, as usually the user would have no way to fix the remote chart on their own. + // + // Given that, we always run `helm dep build` on the chart here, but tolerate any error caused by it + // for a remote chart, so that the user can notice/fix the issue in a local chart while + // a broken remote chart won't completely block their job. + chartPath = normalizeChart(st.basePath, chartPath) if !skipDeps { if err := helm.BuildDeps(release.Name, chartPath); err != nil { - results <- &downloadResults{err: err} - return + if chartFetchedByGoGetter { + diagnostic = fmt.Sprintf( + "WARN: `helm dep build` failed. While processing release %q, Helmfile observed that remote chart %q fetched by go-getter is seemingly broken. "+ + "One of well-known causes of this is that the chart has outdated Chart.lock, which needs the chart maintainer needs to run `helm dep up`. "+ + "Helmfile is tolerating the error to avoid blocking you until the remote chart gets fixed. "+ + "But this may result in any failure later if the chart is broken badly. FYI, the tolerated error was: %v", + release.Name, + chartName, + err.Error(), + ) + } else { + results <- &downloadResults{err: err} + return + } } } + } else if !forceDownload { + // At this point, we are sure that either: + // 1. It is a local chart and we can use it in later process (helm upgrade/template/lint/etc) + // without any modification, or + // 2. It is a remote chart which can be safely handed over to helm, + // because the version of Helm used in this transaction (helm v3 or greater) support downloading + // the chart instead, AND we don't need any modification to the chart + // + // Also see HelmState.chartVersionFlags(). For `helmfile template`, it's called before `helm template` + // only on helm v3. + // For helm 2, we `helm fetch` with the version flags and call `helm template` + // WITHOUT the version flags. } else { - fetchFlags := []string{} - pathElems := []string{ dir, } @@ -911,21 +962,17 @@ chartVersion := "latest" if release.Version != "" { chartVersion = release.Version - fetchFlags = append(fetchFlags, "--version", release.Version) } - pathElems = append(pathElems, release.Name, release.Chart, chartVersion) + pathElems = append(pathElems, release.Name, chartName, chartVersion) chartPath = path.Join(pathElems...) - if st.isDevelopment(release) { - fetchFlags = append(fetchFlags, "--devel") - } - // only fetch chart if it is not already fetched if _, err := os.Stat(chartPath); os.IsNotExist(err) { + fetchFlags := st.chartVersionFlags(release) fetchFlags = append(fetchFlags, "--untar", "--untardir", chartPath) - if err := helm.Fetch(release.Chart, fetchFlags...); err != nil { + if err := helm.Fetch(chartName, fetchFlags...); err != nil { results <- &downloadResults{err: err} return } @@ -938,13 +985,17 @@ } } - results <- &downloadResults{releaseName: release.Name, chartPath: chartPath} + results <- &downloadResults{releaseName: release.Name, chartPath: chartPath, diagnostic: diagnostic} } }, func() { for i := 0; i < len(releases); i++ { downloadRes := <-results + if d := downloadRes.diagnostic; d != "" { + diagnostics = append(diagnostics, d) + } + if downloadRes.err != nil { errs = append(errs, downloadRes.err) @@ -955,6 +1006,11 @@ }, ) + sort.Strings(diagnostics) + for _, d := range diagnostics { + st.logger.Warn(d) + } + if len(errs) > 0 { return nil, errs } @@ -1049,7 +1105,7 @@ } } - if _, err := st.triggerCleanupEvent(release, "template"); err != nil { + if _, err := st.TriggerCleanupEvent(release, "template"); err != nil { st.logger.Warnf("warn: %v\n", err) } } @@ -1125,7 +1181,7 @@ } } - if _, err := st.triggerCleanupEvent(&release, "lint"); err != nil { + if _, err := st.TriggerCleanupEvent(&release, "lint"); err != nil { st.logger.Warnf("warn: %v\n", err) } } @@ -1354,7 +1410,7 @@ } if triggerCleanupEvents { - if _, err := st.triggerCleanupEvent(prep.release, "diff"); err != nil { + if _, err := st.TriggerCleanupEvent(prep.release, "diff"); err != nil { st.logger.Warnf("warn: %v\n", err) } } @@ -1592,7 +1648,7 @@ return st.triggerReleaseEvent("prepare", nil, r, helmfileCommand) } -func (st *HelmState) triggerCleanupEvent(r *ReleaseSpec, helmfileCommand string) (bool, error) { +func (st *HelmState) TriggerCleanupEvent(r *ReleaseSpec, helmfileCommand string) (bool, error) { return st.triggerReleaseEvent("cleanup", nil, r, helmfileCommand) } @@ -1763,14 +1819,7 @@ } func (st *HelmState) flagsForUpgrade(helm helmexec.Interface, release *ReleaseSpec, workerIndex int) ([]string, []string, error) { - flags := []string{} - if release.Version != "" { - flags = append(flags, "--version", release.Version) - } - - if st.isDevelopment(release) { - flags = append(flags, "--devel") - } + flags := st.chartVersionFlags(release) if release.Verify != nil && *release.Verify || release.Verify == nil && st.HelmDefaults.Verify { flags = append(flags, "--verify") @@ -1829,7 +1878,16 @@ } func (st *HelmState) flagsForTemplate(helm helmexec.Interface, release *ReleaseSpec, workerIndex int) ([]string, []string, error) { - flags := []string{} + var flags []string + + // `helm template` in helm v2 does not support `--version` flag. So we fetch with the version flag and then template + // without the flag. See PrepareCharts function to see the Helmfile implementation of chart fetching. + // + // `helm template` in helm v3 supports `--version` and it automatically fetches the remote chart to template, + // so we skip fetching on helmfile-side and let helm fetch it. + if helm.IsHelm3() { + flags = st.chartVersionFlags(release) + } var err error flags, err = st.appendHelmXFlags(flags, release) @@ -1847,14 +1905,7 @@ } func (st *HelmState) flagsForDiff(helm helmexec.Interface, release *ReleaseSpec, workerIndex int) ([]string, []string, error) { - flags := []string{} - if release.Version != "" { - flags = append(flags, "--version", release.Version) - } - - if st.isDevelopment(release) { - flags = append(flags, "--devel") - } + flags := st.chartVersionFlags(release) disableOpenAPIValidation := false if release.DisableOpenAPIValidation != nil { @@ -1893,6 +1944,20 @@ return append(flags, common...), files, nil } +func (st *HelmState) chartVersionFlags(release *ReleaseSpec) []string { + flags := []string{} + + if release.Version != "" { + flags = append(flags, "--version", release.Version) + } + + if st.isDevelopment(release) { + flags = append(flags, "--devel") + } + + return flags +} + func (st *HelmState) appendApiVersionsFlags(flags []string) []string { for _, a := range st.ApiVersions { flags = append(flags, "--api-versions", a) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/pkg/state/state_gogetter_test.go new/helmfile-0.125.7/pkg/state/state_gogetter_test.go --- old/helmfile-0.125.3/pkg/state/state_gogetter_test.go 1970-01-01 01:00:00.000000000 +0100 +++ new/helmfile-0.125.7/pkg/state/state_gogetter_test.go 2020-08-14 12:52:00.000000000 +0200 @@ -0,0 +1,61 @@ +package state + +import ( + "fmt" + "github.com/google/go-cmp/cmp" + "github.com/roboll/helmfile/pkg/helmexec" + "io/ioutil" + "os" + "testing" +) + +func TestGoGetter(t *testing.T) { + logger := helmexec.NewLogger(os.Stderr, "warn") + + testcases := []struct { + chart, dir string + force bool + + out, err string + }{ + { + chart: "raw/incubator", + dir: "", + force: false, + out: "raw/incubator", + err: "", + }, + } + + for i, tc := range testcases { + t.Run(fmt.Sprintf("case %d", i), func(t *testing.T) { + d, err := ioutil.TempDir("", "testgogetter") + if err != nil { + panic(err) + } + defer os.RemoveAll(d) + + st := &HelmState{ + logger: logger, + readFile: ioutil.ReadFile, + basePath: d, + } + + out, err := st.goGetterChart(tc.chart, tc.dir, false) + + if diff := cmp.Diff(tc.out, out); diff != "" { + t.Fatalf("Unexpected out:\n%s", diff) + } + + var errMsg string + + if err != nil { + errMsg = err.Error() + } + + if diff := cmp.Diff(tc.err, errMsg); diff != "" { + t.Fatalf("Unexpected err:\n%s", diff) + } + }) + } +} diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/test/integration/templates-golden/v2/helmx/helmx.all.yaml new/helmfile-0.125.7/test/integration/templates-golden/v2/helmx/helmx.all.yaml --- old/helmfile-0.125.3/test/integration/templates-golden/v2/helmx/helmx.all.yaml 2020-08-06 02:06:25.000000000 +0200 +++ new/helmfile-0.125.7/test/integration/templates-golden/v2/helmx/helmx.all.yaml 1970-01-01 01:00:00.000000000 +0100 @@ -1,17 +0,0 @@ ---- -# Source: helmx/templates/helmx.all.yaml -apiVersion: v1 -data: - namespace: helmx-system -kind: ConfigMap -metadata: - name: release-namespace ---- -apiVersion: v1 -data: - name: helmx -kind: ConfigMap -metadata: - name: release-name - annotations: - foo: bar diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/test/integration/templates-golden/v2/helmx/patched.yaml new/helmfile-0.125.7/test/integration/templates-golden/v2/helmx/patched.yaml --- old/helmfile-0.125.3/test/integration/templates-golden/v2/helmx/patched.yaml 1970-01-01 01:00:00.000000000 +0100 +++ new/helmfile-0.125.7/test/integration/templates-golden/v2/helmx/patched.yaml 2020-08-14 12:52:00.000000000 +0200 @@ -0,0 +1,17 @@ +--- +# Source: helmx/templates/helmx.all.yaml +apiVersion: v1 +data: + namespace: helmx-system +kind: ConfigMap +metadata: + name: release-namespace +--- +apiVersion: v1 +data: + name: helmx +kind: ConfigMap +metadata: + name: release-name + annotations: + foo: bar diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/test/integration/templates-golden/v3/helmx/helmx.all.yaml new/helmfile-0.125.7/test/integration/templates-golden/v3/helmx/helmx.all.yaml --- old/helmfile-0.125.3/test/integration/templates-golden/v3/helmx/helmx.all.yaml 2020-08-06 02:06:25.000000000 +0200 +++ new/helmfile-0.125.7/test/integration/templates-golden/v3/helmx/helmx.all.yaml 1970-01-01 01:00:00.000000000 +0100 @@ -1,18 +0,0 @@ ---- -# Source: helmx/templates/helmx.all.yaml -apiVersion: v1 -data: - namespace: helmx-system -kind: ConfigMap -metadata: - name: release-namespace ---- -# Source: helmx/templates/helmx.all.yaml -apiVersion: v1 -data: - name: helmx -kind: ConfigMap -metadata: - name: release-name - annotations: - foo: bar diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/helmfile-0.125.3/test/integration/templates-golden/v3/helmx/patched.yaml new/helmfile-0.125.7/test/integration/templates-golden/v3/helmx/patched.yaml --- old/helmfile-0.125.3/test/integration/templates-golden/v3/helmx/patched.yaml 1970-01-01 01:00:00.000000000 +0100 +++ new/helmfile-0.125.7/test/integration/templates-golden/v3/helmx/patched.yaml 2020-08-14 12:52:00.000000000 +0200 @@ -0,0 +1,18 @@ +--- +# Source: helmx/templates/helmx.all.yaml +apiVersion: v1 +data: + namespace: helmx-system +kind: ConfigMap +metadata: + name: release-namespace +--- +# Source: helmx/templates/helmx.all.yaml +apiVersion: v1 +data: + name: helmx +kind: ConfigMap +metadata: + name: release-name + annotations: + foo: bar ++++++ vendor.tar.gz ++++++ ++++ 1722 lines of diff (skipped)
