Hello, I'm planning to release the entire setup (based on Terraform) and configuration of the current CI system to a public repository in order give the community the possibility to influence the development and give suggestions. In order to protect credentials, I've considered storing these secrets in Amazon KMS or another similar system while the actual configuration is stored in the publicly accessible repository.
Unfortunately, there are quite a lot of credentials embedded into the Jenkins configuration files. I have created an analysis at https://cwiki.apache.org/confluence/display/MXNET/Proposal%3A+Architecture, see 'Spoiler: File analysis and categorization'. While directories like 'secrets' are clear and *only* contain secrets, most of the other files mix actual configuration with credentials. In general, Jenkins offers a method to define credentials in a central repository so that they can be reference at other locations. Unfortunately, many plugins are not making use of this method and rather store the encrypted value in the configuration files. A bit of background: All plugins use the same mechanism based on hudson.util.Secret (see https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/util/Secret.java). This mechanism is using AES encryption, while the used key is stored at 'secrets/master.key'. This example is part of the configuration for the GitHub SSO plugin, stored in the central Jenkins configuration file config.xml: <securityRealm class="org.jenkinsci.plugins.GithubSecurityRealm"> <githubWebUri>https://github.com</githubWebUri> <githubApiUri>https://api.github.com</githubApiUri> <clientID>1234567890</clientID> <clientSecret>{SomeEncryptedText==}</clientSecret> <oauthScopes>read:org,user:email</oauthScopes> </securityRealm> While it would be desirable to have the central config.xml stored in the public repository, entries like this theoretically classify the file as secret. If you take another look at the analysis, you'll see that this way would classify quite a lot of files as secret and thus render it useless to have the configuration released to the public. But there is one catch: There are only two ways to decrypt these values: 1. Gain access to the file at 'secrets/master.key' 2. Gain the ability to execute the script 'def decrypted = hudson.util.Secret.fromString("{SomeEncryptedText==}").getPlainText()' Luckily, the Jenkins developers are aware of these cases and implemented a few security measurements. First of all, all groovy scripts (aka the Jenkinsfile) are executed in a sandboxed environment and thus unable to access the underlying filesystem. Another method would be to run a job on the master and read the file at 'secrets/master.key', but since we have 0 executor slots available on the master, no user-defined jobs can be executed. Method 1 is thus not possible. So what about the script I've mentioned above? Jenkins restricts the access to certain APIs when executed inside user-defined scripts like the Jenkinsfile. I have tried to access this API by modifying the Jenkinsfile inside a PR at https://github.com/apache/incubator-mxnet/pull/9331 and the result was the following message: "caught org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod hudson.util.Secret fromString java.lang.String" This means that a user is not able to submit a PR to decrypt a secret. The last way to execute this script is the Script-Console, but considering that it is only accessible as an Administrator, I will put this one aside. This means that method 2 is also not possible. Since these are the only two ways to decrypt a secret, we're theoretically safe to publish these encrypted values as long as the master.key stays secret. Other than that, there's no way to decrypt these values. Basically we have to following options: 1. Rely on the encryption and do not hide encrypted values. This means only the actual secrets will be stored in KMS and the entire configuration can be posted to the public repository. This method relies on the ability to keep the master.key a secret - only people with access to the underlying AWS account would be able to access it. 2. Keep the entire Jenkins configuration in a private repository and only grant certain people access. At the moment, this would only be the administrators of the CI (Meghna, Gautam and myself) as well as Chris Olivier due to the constraints mentioned in https://lists.apache.org/thread.html/b28653380fc179a8cc1410d12661a7b1636c09e39a155df157b51bc3@%3Cdev.mxnet.apache.org%3E . What does the community think? Best regards, Marco
