MonkeyCanCode opened a new pull request, #3471: URL: https://github.com/apache/polaris/pull/3471
<!-- ๐ Describe what changes you're proposing, especially breaking or user-facing changes. ๐ See https://github.com/apache/polaris/blob/main/CONTRIBUTING.md for more. --> This PR fixed the following: 1. Remove custom docker regex path in renovate config as that is not necessary and not valid in newer version of renovate 2. Update our renovate config to use the non-deprecated configs to avoid config rewrite (existed one is still valid) 3. Added explanation on why certain dockerfiles are not being monitored by renovate bot Earlier I created https://github.com/apache/polaris/pull/3454 to ensure we have proper regex included for docker. After some testings, I noticed that is actually not necessary and with latest renovate (I was using an older version), those will trigger error as well during configuration validate. Here is the current valication: ``` โ polaris git:(main) docker run --rm \ -v "$PWD:/work" \ renovate/renovate:latest \ renovate-config-validator /work/.github/renovate.json5 INFO: Validating /work/.github/renovate.json5 WARN: Config migration necessary "oldConfig": { "$schema": "https://docs.renovatebot.com/renovate-schema.json";, "extends": [ "config:base", ":preserveSemverRanges", "group:monorepos", "helpers:pinGitHubActionDigests", ":semanticPrefixFixDepsChoreOthers", ":dependencyDashboard" ], "baseBranches": ["main"], "pip_requirements": {"fileMatch": ["(^|/)([\\w-]*)requirements.*\\.txt$"]}, "docker": { "fileMatch": [ "(^|/)Dockerfile$", "(^|/)Dockerfile\\.[^/]*$", "(^|/)Dockerfile-[^/]*$" ] }, "packageRules": [ { "matchManagers": [ "gradle", "gradle-wrapper", "pip_requirements", "pip_setup", "uv", "setup-cfg", "dockerfile", "devcontainer", "docker-compose", "github-actions" ], "automerge": false }, { "matchManagers": ["gradle"], "matchPackagePrefixes": ["software.amazon.awssdk"], "extends": ["schedule:weekly"] }, { "matchManagers": ["pip_requirements", "pip_setup", "uv"], "matchPackagePrefixes": ["boto3"], "extends": ["schedule:weekly"] }, { "groupName": "Quarkus Platform and Group", "matchManagers": ["gradle"], "matchPackageNames": [ "io.quarkus.platform:quarkus-bom", "io.quarkus.platform:quarkus-amazon-services-bom", "io.quarkus.platform:quarkus-google-cloud-services-bom", "io.quarkus:io.quarkus.gradle.plugin" ] }, {"matchPackageNames": ["openapi-generator-cli"], "enabled": false} ], "prConcurrentLimit": 50, "prHourlyLimit": 5, "labels": ["renovate-polaris"] }, "newConfig": { "$schema": "https://docs.renovatebot.com/renovate-schema.json";, "extends": [ "config:recommended", ":preserveSemverRanges", "group:monorepos", "helpers:pinGitHubActionDigests", ":semanticPrefixFixDepsChoreOthers", ":dependencyDashboard" ], "baseBranchPatterns": ["main"], "pip_requirements": { "managerFilePatterns": ["/(^|/)([\\w-]*)requirements.*\\.txt$/"] }, "packageRules": [ { "matchCategories": ["docker"], "managerFilePatterns": [ "/(^|/)Dockerfile$/", "/(^|/)Dockerfile\\.[^/]*$/", "/(^|/)Dockerfile-[^/]*$/" ] }, { "matchManagers": [ "gradle", "gradle-wrapper", "pip_requirements", "pip_setup", "uv", "setup-cfg", "dockerfile", "devcontainer", "docker-compose", "github-actions" ], "automerge": false }, { "matchManagers": ["gradle"], "extends": ["schedule:weekly"], "matchPackageNames": ["software.amazon.awssdk{/,}**"] }, { "matchManagers": ["pip_requirements", "pip_setup", "uv"], "extends": ["schedule:weekly"], "matchPackageNames": ["boto3{/,}**"] }, { "groupName": "Quarkus Platform and Group", "matchManagers": ["gradle"], "matchPackageNames": [ "io.quarkus.platform:quarkus-bom", "io.quarkus.platform:quarkus-amazon-services-bom", "io.quarkus.platform:quarkus-google-cloud-services-bom", "io.quarkus:io.quarkus.gradle.plugin" ] }, {"matchPackageNames": ["openapi-generator-cli"], "enabled": false} ], "prConcurrentLimit": 50, "prHourlyLimit": 5, "labels": ["renovate-polaris"] } WARN: Found errors in configuration "file": "/work/.github/renovate.json5", "warnings": [ { "topic": "packageRules[0].managerFilePatterns", "message": "\"managerFilePatterns\" can't be used in \"packageRules\". Allowed objects: ansible, ansible-galaxy, argocd, asdf, azure-pipelines, batect, batect-wrapper, bazel, bazel-module, bazelisk, bicep, bitbucket-pipelines, bitrise, buildkite, buildpacks, bun, bun-version, bundler, cake, cargo, cdnurl, circleci, cloudbuild, cocoapods, composer, conan, copier, cpanfile, crossplane, crow, customManagers, deps-edn, devbox, devcontainer, docker-compose, dockerfile, droneci, fleet, flux, fvm, git-submodules, github-actions, gitlabci, gitlabci-include, glasskube, gleam, gomod, gradle, gradle-wrapper, haskell-cabal, helm-requirements, helm-values, helmfile, helmsman, helmv3, hermit, homebrew, html, jenkins, jsonnet-bundler, kotlin-script, kubernetes, kustomize, leiningen, maven, maven-wrapper, meteor, mint, mise, mix, nix, nodenv, npm, nuget, nvm, ocb, osgi, pep621, pep723, pip-compile, pip_requirements, pip_setup, pipenv, pixi, poetry, pre-commit, pub, puppet, pyenv, quadl et, renovate-config-presets, ruby-version, runtime-version, sbt, scalafmt, setup-cfg, sveltos, swift, tekton, terraform, terraform-version, terragrunt, terragrunt-version, tflint-plugin, travis, typst, unity3d, velaci, vendir, woodpecker." } ] ``` The error is actually from the docker section which is no longer valid for newer version of renovate. Here is the output for the fixed with this PR: ``` โ polaris git:(fix_renovate) docker run --rm \ -v "$PWD:/work" \ renovate/renovate:latest \ renovate-config-validator /work/.github/renovate.json5 INFO: Validating /work/.github/renovate.json5 INFO: Config validated successfully ``` Now back to the original problem where we have couple dockerfiles which are not getting updated. Those are actually done by renovate itself. Here is what I found: the default regex used by renovate for docker is following (ref: https://docs.renovatebot.com/modules/manager/dockerfile/): ``` { "managerFilePatterns": [ "/(^|/|\\.)([Dd]ocker|[Cc]ontainer)file$/", "/(^|/)([Dd]ocker|[Cc]ontainer)file[^/]*$/" ] } ``` with above, it should cover the ones we we have as it include `dockerfile*|Dockerfile*|containerfile*|Containerfile*`. However, this doesn't check certain ones due to `config:base` (now `config:recommended`) which has certain dirs excluded (ref: https://docs.renovatebot.com/presets-default/#ignoremodulesandtests). In our case, it excluded all of the dockerfile under paths `*/test/*|`: ``` monitored: ./plugins/spark/v3.5/regtests/Dockerfile ./plugins/spark/v3.5/getting-started/notebooks/Dockerfile ./runtime/admin/src/testFixtures/resources/org/apache/polaris/admintool/Dockerfile-postgres-version ./runtime/admin/src/main/docker/Dockerfile.jvm ./runtime/server/src/main/docker/Dockerfile.jvm ./regtests/Dockerfile ./getting-started/spark/notebooks/Dockerfile ./persistence/nosql/persistence/db/mongodb/src/testFixtures/resources/org/apache/polaris/persistence/nosql/mongodb/Dockerfile-mongodb-version ./site/docker/Dockerfile not monitored: ./tools/minio-testcontainer/src/main/resources/org/apache/polaris/test/minio/Dockerfile-minio-version ./runtime/test-common/src/main/resources/org/apache/polaris/test/commons/s3mock/Dockerfile-s3mock-version ./runtime/test-common/src/main/resources/org/apache/polaris/test/commons/Dockerfile-postgres-version ./runtime/test-common/src/main/resources/org/apache/polaris/test/commons/keycloak/Dockerfile-keycloak-version ./runtime/service/src/test/resources/org/apache/polaris/service/events/jsonEventListener/aws/cloudwatch/Dockerfile-localstack-version ./extensions/auth/opa/tests/src/intTest/resources/org/apache/polaris/extension/auth/opa/test/Dockerfile-opa-version ``` That being said, to be able to have renovate run on all dockerfiles, we will need to update `ignorePaths` for dockerfile. Here is the fixed output: ``` DEBUG: Matched 15 file(s) for manager dockerfile: extensions/auth/opa/tests/src/intTest/resources/org/apache/polaris/extension/auth/opa/test/Dockerfile-opa-version, getting-started/spark/notebooks/Dockerfile, persistence/nosql/persistence/db/mongodb/src/testFixtures/resources/org/apache/polaris/persistence/nosql/mongodb/Dockerfile-mongodb-version, plugins/spark/v3.5/getting-started/notebooks/Dockerfile, plugins/spark/v3.5/regtests/Dockerfile, regtests/Dockerfile, runtime/admin/src/main/docker/Dockerfile.jvm, runtime/admin/src/testFixtures/resources/org/apache/polaris/admintool/Dockerfile-postgres-version, runtime/server/src/main/docker/Dockerfile.jvm, runtime/service/src/test/resources/org/apache/polaris/service/events/jsonEventListener/aws/cloudwatch/Dockerfile-localstack-version, runtime/test-common/src/main/resources/org/apache/polaris/test/commons/Dockerfile-postgres-version, runtime/test-common/src/main/resources/org/apache/polaris/test/commons/keycloak/Dockerfile-keycloak-ve rsion, runtime/test-common/src/main/resources/org/apache/polaris/test/commons/s3mock/Dockerfile-s3mock-version, site/docker/Dockerfile, tools/minio-testcontainer/src/main/resources/org/apache/polaris/test/minio/Dockerfile-minio-version (repository=local) ``` Here is earlier: ``` DEBUG: Matched 9 file(s) for manager dockerfile: getting-started/spark/notebooks/Dockerfile, persistence/nosql/persistence/db/mongodb/src/testFixtures/resources/org/apache/polaris/persistence/nosql/mongodb/Dockerfile-mongodb-version, plugins/spark/v3.5/getting-started/notebooks/Dockerfile, plugins/spark/v3.5/regtests/Dockerfile, regtests/Dockerfile, runtime/admin/src/main/docker/Dockerfile.jvm, runtime/admin/src/testFixtures/resources/org/apache/polaris/admintool/Dockerfile-postgres-version, runtime/server/src/main/docker/Dockerfile.jvm, site/docker/Dockerfile (repository=local) ``` ## Checklist - [x] ๐ก๏ธ Don't disclose security issues! (contact [email protected]) - [x] ๐ Clearly explained why the changes are needed, or linked related issues: Fixes # - [x] ๐งช Added/updated tests with good coverage, or manually tested (and explained how) - [x] ๐ก Added comments for complex logic - [x] ๐งพ Updated `CHANGELOG.md` (if needed) - [x] ๐ Updated documentation in `site/content/in-dev/unreleased` (if needed) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
