Thanks Jonathan and Gianluca for your explanations. I now got most of my confusion cleared up, most importantly the fact that String interpolation is done either by Groovy or by the bash but never by Jenkins.
I think the first sentence here got me on the wrong path: "Jenkins Pipeline uses rules identical to Groovy <http://groovy-lang.org/> for string interpolation" https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#string-interpolation To me that suggests that Jenkins also does some kind of string interpolation. @Gianluca: Thanks for your attempts to solve my problem. I've tried again, and I am pretty sure now that mixing simple strings (single quotes) and GStrings (double quotes) does not help because I suspect that there is some automatic type conversion between simple strings and GStrings involved here that destroys the simple strings - see below. Also keep in mind that I need to quote all of jdbcUrl/username/password in the bash command because they can contain special characters (& ! etc). So trying your approach like this: sh "${mvnCommand(m2RepoIdentifier)}" + ' -f proj-reactor/pom.xml ' + ' -P GenerateStorables -DdeployAtEnd=true' + " '-Dproj.build.generatefromdb.url=${jdbcUrlNoData()}'" + ' -Dproj.build.generatefromdb.user=${GEN_USR}' + ' "-Dproj.build.generatefromdb.password=${GEN_PASSWD}"' + " clean ${mvnBuildGoal} pmd:pmd pmd:cpd" -> Jenkins *will still print the warning* about insecure interpolation of sensitive variables -> The build works fine though, e.g. all variables get passed to the maven build as expected *BUT what actually worked* in the end is the following: sh "${mvnCommand(m2RepoIdentifier)}".toString() + ' -f proj-reactor/pom.xml ' + ' -P GenerateStorables -DdeployAtEnd=true' + " '-Dproj.build.generatefromdb.url=${jdbcUrlNoData()}'".toString() + ' -Dproj.build.generatefromdb.user=${GEN_USR}' + ' "-Dproj.build.generatefromdb.password=${GEN_PASSWD}"' + " clean ${mvnBuildGoal} pmd:pmd pmd:cpd".toString() -> No warning about insecure interpolation of sensitive variables -> The build works fine, e.g. all variables get passed to the maven build as expected -> Also nice for readability to not need to escape any quotes/characters in any part of the command above Hope this helps other folks to clear up any string interpolation confusion :-) stefan. On Wed, Nov 18, 2020 at 1:14 AM JonathanRRogers <[email protected]> wrote: > Unless "mvnBuildGoal" is the name of a Jenkins environment variable, which > would be added to the shell process environment, it won't work inside a > single-quoted string. > > On Tuesday, November 17, 2020 at 4:11:32 PM UTC-5 Gianluca wrote: > >> Somethings said are correctly … somethings said are wrong. >> I’ll try to summarise pointing to the lines in your pipeline and explain >> what’s wrong and how to fix: >> >> Single quoted string are literals. They never get interpolated and passed >> as they are. >> So, in your case: >> >> sh '${JAVA_HOME}/bin/java -version' // single quotes - this works fine! >>> >> >> Work … not because ${JAVA_HOME} is interpolated by Jenkins … but because >> for an unfortunate coincidence … ${JAVA_HOME} is a correct bash syntax for >> reference an environment variable in bash and get expanded correctly. >> DO NOT confuse Jenkins pipeline environment variable that exists only in >> the pipeline context with the BASH environment variable that exists in the >> shell when “sh” is executed. >> Now, also take into account that Jenkins also creates a corresponding >> BASH environment variable for each pipeline environment variable. >> So, what in Groovy double-quoted string like “${env.GEN_USR}” get >> expanded using the value of the Jenkins environment variable GEN_USR, but >> if you want to let bash expand the environment variable then you have to >> pass '${GEN_USR}’ … in single-quoted string … or … you need to escape “$” >> in double-quoted string to avoid that Groovy does the interpolation instead >> of bash. >> >> Your multi-line command suffer from the fact somethings need to be >> expanded by Groovy and something else need to be expanded by BASH, so you >> need to split in different string and concatenate or escape $: >> >> sh script: ''' >>> ${mvnCommand(m2RepoIdentifier)}\ >>> >> This is a Jenkins function … so need to be expanded by Groovy and it’s in >> the double-quoted string… so far so good >> >> -f proj-reactor/pom.xml\ >>> -P GenerateStorables -DdeployAtEnd=true\ >>> -Dproj.build.generatefromdb.url=${jdbcUrlNoData()}\ >>> >> That is the same … so it’s good >> >> -Dproj.build.generatefromdb.user=${GEN_USR}\ >>> >> That’s the problem. You are using “withCredentials” and that command only >> creates BASH environment variable … and you are letting Groovy do the >> expansion but GEN_USR doesn’t exist into Jenkins pipeline … and neither >> env.GEN_USR … the only one that exist is the BASH environment variable and >> you have to let bash expand … so you need to escape $ to pass as it is to >> BASH: >> >> -Dproj.build.generatefromdb.user=\${GEN_USR}\ >> >> -Dproj.build.generatefromdb.password=${GEN_PASSWD}\ >>> >> Same as above… it’s a BASH environment variable >> >> clean ${mvnBuildGoal} pmd:pmd pmd:cpd\ >>> >> That is probably a Groovy variable but I’m not entire sure. >> >> ''' >>> >> >> I hope that helps. >> >> As suggestion, try to split and use a mix of double-quoted and >> single-quoted so you avoid escaping: >> >> sh script: >> "${mvnCommand(m2RepoIdentifier)} “ + >> '-f proj-reactor/pom.xml ‘ + >> '-P GenerateStorables -DdeployAtEnd=true ‘ + >> "-Dproj.build.generatefromdb.url=${jdbcUrlNoData()}” + >> '-Dproj.build.generatefromdb.user=${GEN_USR} ‘ + >> '-Dproj.build.generatefromdb.password=${GEN_PASSWD} ‘ + >> 'clean ${mvnBuildGoal} pmd:pmd pmd:cpd' >> >> >> Cheers, >> Gianluca. >> >> >> On 17 Nov 2020, at 20:45, JonathanRRogers <[email protected]> wrote: >> >> I don't think you've read the Jenkins pipeline documentation correctly. >> String interpolation is performed by Groovy. Jenkins doesn't add any >> interpolation behavior. Jenkins will always treat single-quoted strings as >> simple literals. The "sh" step calls a shell which can do its own variable >> expansion, which is the approach recommended for passing credentials to >> external commands. >> >> Your multi-line sh step includes calls to "mvnCommand()" and >> "jdbcUrlNoData()". Unless those are shell functions, you're unlikely to get >> the results you want. >> >> On Tuesday, November 17, 2020 at 10:41:51 AM UTC-5 [email protected] >> wrote: >> >>> Thanks Dirk, but that does not solve my problem, I'm still stuck. >>> I think that my question boils down to why the multi-line sh command >>> below does not work? >>> According to >>> https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#string-interpolation >>> single-quoted strings should be interpolated by Jenkins (instead of >>> using Groovy string interpolation). I would assume this applies >>> to both variables (e.g. GEN_USR below) and method calls returning a >>> string (e.g. jdbcUrlNoData() below)? >>> >>> withCredentials([usernamePassword(credentialsId: >>> 'credentialsGenerateFromDb', >>> usernameVariable: 'GEN_USR', >>> passwordVariable: 'GEN_PASSWD')]) { >>> String m2RepoIdentifier = "${env.BRANCH_NAME}_${env.BUILD_NUMBER}" >>> >>> withEnv(javaAndMavenEnvArray()) { >>> sh '${JAVA_HOME}/bin/java -version' // single quotes - this works >>> fine! >>> >>> sh script: ''' >>> ${mvnCommand(m2RepoIdentifier)}\ >>> -f proj-reactor/pom.xml\ >>> -P GenerateStorables -DdeployAtEnd=true\ >>> -Dproj.build.generatefromdb.url=${jdbcUrlNoData()}\ >>> -Dproj.build.generatefromdb.user=${GEN_USR}\ >>> -Dproj.build.generatefromdb.password=${GEN_PASSWD}\ >>> clean ${mvnBuildGoal} pmd:pmd pmd:cpd\ >>> ''' >>> } >>> } >>> >>> On Tue, Nov 17, 2020 at 2:06 PM 'Dirk Heinrichs' via Jenkins Users < >>> [email protected]> wrote: >>> >>>> Am Dienstag, den 17.11.2020, 13:45 +0100 schrieb ST: >>>> >>>> " '-Dproj.build.generatefromdb.user=${env.GEN_USR}'" >>>> >>>> >>>> Try with ${GEN_USR} inside shell scripts, not ${env.GEN_USR}. >>>> >>>> HTH... >>>> >>>> Dirk >>>> >>>> -- >>>> >>>> *Dirk Heinrichs* >>>> Senior Systems Engineer, Delivery Pipeline >>>> OpenText ™ Discovery | Recommind >>>> *Phone*: +49 2226 15966 18 <+49%202226%201596618> >>>> *Email*: [email protected] >>>> *Website*: www.recommind.de >>>> Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach >>>> Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu >>>> Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, >>>> Registernummer HRB 10646 >>>> This e-mail may contain confidential and/or privileged information. If >>>> you are not the intended recipient (or have received this e-mail in error) >>>> please notify the sender immediately and destroy this e-mail. Any >>>> unauthorized copying, disclosure or distribution of the material in this >>>> e-mail is strictly forbidden >>>> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte >>>> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail >>>> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und >>>> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte >>>> Weitergabe dieser Mail sind nicht gestattet. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-users/4dd32ea372d43c9dde0800bed9a16253d45cb2e5.camel%40opentext.com >>>> <https://groups.google.com/d/msgid/jenkinsci-users/4dd32ea372d43c9dde0800bed9a16253d45cb2e5.camel%40opentext.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/bed31997-28f8-4e9b-b205-43f02d4348a5n%40googlegroups.com >> <https://groups.google.com/d/msgid/jenkinsci-users/bed31997-28f8-4e9b-b205-43f02d4348a5n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/b5086225-d400-4500-a936-0413c025edf6n%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-users/b5086225-d400-4500-a936-0413c025edf6n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CABwQARsOttX5sj2E0BwZ%2Bx8xc6R35Qy1gS1OvYhE4ghpzN%3D%2BkQ%40mail.gmail.com.
