The error you see seems to happen because when validating secret lookup references (to check the *rules *configured on the secret config) the logic seems to incorrectly assume that any ScmMaterial with secrets will have 1+ pipeline using that secret. It then incorrectly assumes that "all pipelines have secret lookup errors" when it compares # pipelines and # pipelines with errors (0 == 0, oops) and fails.
That's why you get the useless/empty error message and broken resolution. If you turn on DEBUG logging for *com.thoughtworks.go.server.materials.MaterialDatabaseUpdater* you'll see the below with no error message: 2024-11-03 00:14:00,477 DEBUG [147@MessageListener for MaterialUpdateListener] MaterialDatabaseUpdater:128 - [Material Update] Modification check failed for material: URL: https://github.com/gocd/aws, Branch: master com.thoughtworks.go.server.exceptions.RulesViolationException: at com.thoughtworks.go.server.service.RulesService.validateSecretConfigReferences(RulesService.java:82) at com.thoughtworks.go.server.service.SecretParamResolver.resolve(SecretParamResolver.java:77) at com.thoughtworks.go.server.service.SecretParamResolver.resolve(SecretParamResolver.java:67) at com.thoughtworks.go.server.service.MaterialService.resolveSecretParams(MaterialService.java:163) at com.thoughtworks.go.server.service.MaterialService.latestModification(MaterialService.java:126) at com.thoughtworks.go.server.materials.LegacyMaterialChecker.findLatestModification(LegacyMaterialChecker.java:50) at com.thoughtworks.go.server.materials.ScmMaterialUpdater.insertLatestOrNewModifications(ScmMaterialUpdater.java:55) at com.thoughtworks.go.server.materials.ScmMaterialUpdater.addNewMaterialWithModifications(ScmMaterialUpdater.java:70) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.addNewMaterialWithModifications(MaterialDatabaseUpdater.java:179) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.initializeMaterialWithLatestRevision(MaterialDatabaseUpdater.java:137) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater$1.doInTransaction(MaterialDatabaseUpdater.java:95) at com.thoughtworks.go.server.transaction.TransactionCallback.doWithExceptionHandling(TransactionCallback.java:23) at com.thoughtworks.go.server.transaction.TransactionTemplate.lambda$executeWithExceptionHandling$1(TransactionTemplate.java:43) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133) at com.thoughtworks.go.server.transaction.TransactionTemplate.executeWithExceptionHandling(TransactionTemplate.java:40) at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterial(MaterialDatabaseUpdater.java:92) at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:64) at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:32) at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83) at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63) at java.base/java.lang.Thread.run(Thread.java:1583) If I fix the error handling so it doesn't incorrectly consider it a rules violation it seems to work OK. However I'll need to think a little bit more about whether it *should*. :-) I think if you add at least 1 pipeline that refers to the same logical material (same type, URL, branch, username) you might find it works as expected? *Test Connection* works because it seems to resolve secrets without validating the references in the same way .... for better or worse. Arguably it should be validating references here also. -Chad On Fri, Nov 1, 2024 at 1:10 AM Chad Wilson <ch...@thoughtworks.com> wrote: > Not really, other than to raise a proper bug/enhancement report at > https://github.com/gocd/gocd/issues preferably with the easiest way to > replicate this so it can be tracked. I'd be digging into the GoCD issue > history to see if this was expected to work (or worked at some point) the > same as you :-) > > -Chad > > On Thu, Oct 31, 2024 at 10:32 PM Jason Smyth <jsm...@taqauto.com> wrote: > >> Hi Chad, >> >> Here is the error as it appears in the GoCD UI: >> >> [image: 2024-10-31 10-21 - brave_hDTIkVSRsL.png] >> >> I tried the "GoCD file based Secrets Plugin" and replicated the issue >> with that secret manager as well. I think this suggests that the issue is >> in the core GoCD configuration repo module. Unless there is something wrong >> with my test cases. >> >> Thoughts? >> >> Regards, >> Jason Smyth >> >> >> On Wednesday 30 October 2024 at 10:24:45 UTC-4 Jason Smyth wrote: >> >>> Hi Chad, >>> >>> >>> >>> This is on the latest (I believe) version: v24.3.0. >>> >>> >>> >>> Regards, >>> >>> *Jason Smyth* >>> >>> >>> >>> *From:* go...@googlegroups.com <go...@googlegroups.com> *On Behalf Of *Chad >>> Wilson >>> *Sent:* Tuesday, October 29, 2024 11:00 PM >>> *To:* go...@googlegroups.com >>> *Subject:* Re: [go-cd] Config Repository Not Parsing When Credentials >>> Supplied via Secret (v24.3.0) >>> >>> >>> >>> OK, thanks. >>> >>> Are you working with a "modern" GoCD version, or is this still some >>> older 19.x version? >>> >>> >>> >>> -Chad >>> >>> >>> >>> On Wed, Oct 30, 2024 at 5:28 AM Jason Smyth <jsm...@taqauto.com> wrote: >>> >>> Hi Chad, >>> >>> >>> >>> My tests were done using the "AWS Secrets Manager plugin for GoCD". >>> There is nothing in go-server.log aside from what I posted in my initial >>> message, and the secret plugin logs are empty. I can try turning up logging >>> verbosity and repeating the tests, but I'm not sure how to do that in my >>> Docker Compose test environment. >>> >>> >>> >>> I know that the secret plugin is working. I have pipelines configured in >>> a working (because I used a password) config repo, and those pipelines >>> depend on the same secret plugin. Additionally, the test connection button >>> in the config repo modal returns errors if I intentionally misconfigure the >>> secret. >>> >>> >>> >>> I don't think it has anything to do with the repository contents. I have >>> replicated the behaviour in 2 different test systems, with multiple source >>> repos, including some that are known to be good because they are accessed >>> by other GoCD instances (albeit without secrets plugin references). I have >>> also replicated the issue with both JSON and YAML repos, so I don't think >>> the issue is with the individual configuration repository plugins. >>> >>> >>> >>> The config repos are not used as additional materials. I tested adding a >>> config repo that pointed to a repo that _is_ used as a material for >>> pipelines, and it works properly. I suspect that this means that the >>> existing material configuration takes precedence and GoCD doesn't bother >>> trying to re-clone, but I could be mistaken there. >>> >>> >>> >>> If I get a chance, I will try to see if I can replicate the issue with >>> the "GoCD file based Secrets Plugin". That should provide some indication >>> of whether the issue is in the secrets plugin or the core GoCD >>> configuration repo module. >>> >>> >>> >>> Any other thoughts on things to try? >>> >>> >>> >>> Regards, >>> >>> Jason Smyth >>> >>> >>> >>> On Tuesday 22 October 2024 at 02:42:08 UTC-4 Chad Wilson wrote: >>> >>> This sounds weird/unexpected but haven't had time to try reproducing >>> this myself. Which secrets plugin are you using? What's in the logs for the >>> server or the secret plugin itself? >>> >>> >>> >>> Is the same repo url also used for other materials (whether config repos >>> or normal materials)? >>> >>> On Fri, 18 Oct 2024, 23:29 Jason Smyth, <jsm...@taqauto.com> wrote: >>> >>> Hello community, >>> >>> >>> >>> I encountered a strange issue whereby config repositories don’t seem to >>> work properly when we try to supply the password via a secret instead of >>> directly in the config repo config. The connection test succeeds, so the >>> system is fetching the password from the secret at that point, but once >>> saved, the config repo fails to parse. >>> >>> >>> >>> The error message (URL redacted) I see in the UI is: >>> >>> >>> >>> There was an error parsing this configuration repository: >>> >>> MODIFICATION CHECK FAILED FOR MATERIAL: URL: >>> HTTPS://DEV.AZURE.COM/ORGANIZATION/TEAMPROJECT/_GIT/SANDBOX-JASONS, >>> BRANCH: GOCD-PIPELINE-TEST >>> >>> NO PIPELINES ARE AFFECTED BY THIS MATERIAL, PERHAPS THIS MATERIAL IS >>> UNUSED. >>> >>> >>> >>> Failed to load pipelines defined in this repository: There was an >>> unknown error performing the operation. Possible reason (Not Found) >>> >>> >>> >>> The GoCD server log shows the following warning: >>> >>> >>> >>> 2024-10-18 10:47:01 jvm 1 | 2024-10-18 14:47:01,263 WARN >>> [143@MessageListener for ConfigMaterialUpdateListener] >>> ConfigMaterialUpdateListener:65 - [Config Material Update] Cannot update >>> configuration part because material update has failed. Reason: >>> >>> >>> >>> When I switched from using a secret to directly supplying the password >>> via the UI, the configuration repository started working as intended. >>> >>> >>> >>> I’m reasonably certain that this is a bug, but wanted to check with the >>> community to confirm that using secrets in this way is supposed to be a >>> supported use-case. >>> >>> >>> >>> Any thoughts or guidance would be appreciated. >>> >>> >>> >>> Regards, >>> >>> *Jason Smyth* >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to go-cd+un...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/DM6PR16MB36715673328A38D68EA7AE8ECF402%40DM6PR16MB3671.namprd16.prod.outlook.com >>> <https://groups.google.com/d/msgid/go-cd/DM6PR16MB36715673328A38D68EA7AE8ECF402%40DM6PR16MB3671.namprd16.prod.outlook.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to go-cd+un...@googlegroups.com. >>> To view this discussion visit >>> https://groups.google.com/d/msgid/go-cd/5e1c44b3-be46-4764-8a39-fb45703f9193n%40googlegroups.com >>> <https://groups.google.com/d/msgid/go-cd/5e1c44b3-be46-4764-8a39-fb45703f9193n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "go-cd" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/go-cd/JlzHTa-Vy_0/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> go-cd+un...@googlegroups.com. >>> To view this discussion visit >>> https://groups.google.com/d/msgid/go-cd/CAA1RwH-qHq8U1qXRiZuupuh-T0ojU8j%3DTSbAt4VveEahAP8DMg%40mail.gmail.com >>> <https://groups.google.com/d/msgid/go-cd/CAA1RwH-qHq8U1qXRiZuupuh-T0ojU8j%3DTSbAt4VveEahAP8DMg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to go-cd+unsubscr...@googlegroups.com. >> To view this discussion visit >> https://groups.google.com/d/msgid/go-cd/8b6f8c9a-78fd-45c2-badb-3a46befff0dcn%40googlegroups.com >> <https://groups.google.com/d/msgid/go-cd/8b6f8c9a-78fd-45c2-badb-3a46befff0dcn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAA1RwH-hKAhXaXbqeZRfYNoEmCXfT71jCtQxvw%3D4eYynCDbHTA%40mail.gmail.com.
