I'm glad you figured it out. It sure looks like a mess. On Wednesday, November 18, 2020 at 5:02:41 PM UTC-5 [email protected] wrote:
> Thanks Jonathan and Gianluca for your explanations. I now got most of my > confusion cleared up, most importantly the fact that String interpolation > is done either by Groovy or by the bash but never by Jenkins. > > I think the first sentence here got me on the wrong path: "Jenkins > Pipeline uses rules identical to Groovy <http://groovy-lang.org/> for > string interpolation" > https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#string-interpolation > To me that suggests that Jenkins also does some kind of string > interpolation. > > @Gianluca: Thanks for your attempts to solve my problem. I've tried again, > and I am pretty sure now that mixing simple strings (single quotes) and > GStrings (double quotes) does not help because I suspect that there is some > automatic type conversion between simple strings and GStrings involved here > that destroys the simple strings - see below. > > Also keep in mind that I need to quote all of jdbcUrl/username/password in > the bash command because they can contain special characters (& ! etc). > > So trying your approach like this: > sh "${mvnCommand(m2RepoIdentifier)}" + > > ' -f proj-reactor/pom.xml ' + > ' -P GenerateStorables -DdeployAtEnd=true' + > " '-Dproj.build.generatefromdb.url=${jdbcUrlNoData()}'" + > ' -Dproj.build.generatefromdb.user=${GEN_USR}' + > ' "-Dproj.build.generatefromdb.password=${GEN_PASSWD}"' + > " clean ${mvnBuildGoal} pmd:pmd pmd:cpd" > -> Jenkins *will still print the warning* about insecure interpolation of > sensitive variables > -> The build works fine though, e.g. all variables get passed to the maven > build as expected > > *BUT what actually worked* in the end is the following: > sh "${mvnCommand(m2RepoIdentifier)}".toString() + > > ' -f proj-reactor/pom.xml ' + > ' -P GenerateStorables -DdeployAtEnd=true' + > " '-Dproj.build.generatefromdb.url=${jdbcUrlNoData()}'".toString() + > > ' -Dproj.build.generatefromdb.user=${GEN_USR}' + > ' "-Dproj.build.generatefromdb.password=${GEN_PASSWD}"' + > " clean ${mvnBuildGoal} pmd:pmd pmd:cpd".toString() > -> No warning about insecure interpolation of sensitive variables > -> The build works fine, e.g. all variables get passed to the maven build > as expected > -> Also nice for readability to not need to escape any quotes/characters > in any part of the command above > > Hope this helps other folks to clear up any string interpolation confusion > :-) > stefan. > > > On Wed, Nov 18, 2020 at 1:14 AM JonathanRRogers <[email protected]> > wrote: > >> Unless "mvnBuildGoal" is the name of a Jenkins environment variable, >> which would be added to the shell process environment, it won't work inside >> a single-quoted string. >> >> On Tuesday, November 17, 2020 at 4:11:32 PM UTC-5 Gianluca wrote: >> >>> Somethings said are correctly … somethings said are wrong. >>> I’ll try to summarise pointing to the lines in your pipeline and explain >>> what’s wrong and how to fix: >>> >>> Single quoted string are literals. They never get interpolated and >>> passed as they are. >>> So, in your case: >>> >>> sh '${JAVA_HOME}/bin/java -version' // single quotes - this works fine! >>>> >>> >>> Work … not because ${JAVA_HOME} is interpolated by Jenkins … but because >>> for an unfortunate coincidence … ${JAVA_HOME} is a correct bash syntax for >>> reference an environment variable in bash and get expanded correctly. >>> DO NOT confuse Jenkins pipeline environment variable that exists only in >>> the pipeline context with the BASH environment variable that exists in the >>> shell when “sh” is executed. >>> Now, also take into account that Jenkins also creates a corresponding >>> BASH environment variable for each pipeline environment variable. >>> So, what in Groovy double-quoted string like “${env.GEN_USR}” get >>> expanded using the value of the Jenkins environment variable GEN_USR, but >>> if you want to let bash expand the environment variable then you have to >>> pass '${GEN_USR}’ … in single-quoted string … or … you need to escape “$” >>> in double-quoted string to avoid that Groovy does the interpolation instead >>> of bash. >>> >>> Your multi-line command suffer from the fact somethings need to be >>> expanded by Groovy and something else need to be expanded by BASH, so you >>> need to split in different string and concatenate or escape $: >>> >>> sh script: ''' >>>> ${mvnCommand(m2RepoIdentifier)}\ >>>> >>> This is a Jenkins function … so need to be expanded by Groovy and it’s >>> in the double-quoted string… so far so good >>> >>> -f proj-reactor/pom.xml\ >>>> -P GenerateStorables -DdeployAtEnd=true\ >>>> -Dproj.build.generatefromdb.url=${jdbcUrlNoData()}\ >>>> >>> That is the same … so it’s good >>> >>> -Dproj.build.generatefromdb.user=${GEN_USR}\ >>>> >>> That’s the problem. You are using “withCredentials” and that command >>> only creates BASH environment variable … and you are letting Groovy do the >>> expansion but GEN_USR doesn’t exist into Jenkins pipeline … and neither >>> env.GEN_USR … the only one that exist is the BASH environment variable and >>> you have to let bash expand … so you need to escape $ to pass as it is to >>> BASH: >>> >>> -Dproj.build.generatefromdb.user=\${GEN_USR}\ >>> >>> -Dproj.build.generatefromdb.password=${GEN_PASSWD}\ >>>> >>> Same as above… it’s a BASH environment variable >>> >>> clean ${mvnBuildGoal} pmd:pmd pmd:cpd\ >>>> >>> That is probably a Groovy variable but I’m not entire sure. >>> >>> ''' >>>> >>> >>> I hope that helps. >>> >>> As suggestion, try to split and use a mix of double-quoted and >>> single-quoted so you avoid escaping: >>> >>> sh script: >>> "${mvnCommand(m2RepoIdentifier)} “ + >>> '-f proj-reactor/pom.xml ‘ + >>> '-P GenerateStorables -DdeployAtEnd=true ‘ + >>> "-Dproj.build.generatefromdb.url=${jdbcUrlNoData()}” + >>> '-Dproj.build.generatefromdb.user=${GEN_USR} ‘ + >>> '-Dproj.build.generatefromdb.password=${GEN_PASSWD} ‘ + >>> 'clean ${mvnBuildGoal} pmd:pmd pmd:cpd' >>> >>> >>> Cheers, >>> Gianluca. >>> >>> >>> On 17 Nov 2020, at 20:45, JonathanRRogers <[email protected]> wrote: >>> >>> I don't think you've read the Jenkins pipeline documentation correctly. >>> String interpolation is performed by Groovy. Jenkins doesn't add any >>> interpolation behavior. Jenkins will always treat single-quoted strings as >>> simple literals. The "sh" step calls a shell which can do its own variable >>> expansion, which is the approach recommended for passing credentials to >>> external commands. >>> >>> Your multi-line sh step includes calls to "mvnCommand()" and >>> "jdbcUrlNoData()". Unless those are shell functions, you're unlikely to get >>> the results you want. >>> >>> On Tuesday, November 17, 2020 at 10:41:51 AM UTC-5 [email protected] >>> wrote: >>> >>>> Thanks Dirk, but that does not solve my problem, I'm still stuck. >>>> I think that my question boils down to why the multi-line sh command >>>> below does not work? >>>> According to >>>> https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#string-interpolation >>>> single-quoted strings should be interpolated by Jenkins (instead of >>>> using Groovy string interpolation). I would assume this applies >>>> to both variables (e.g. GEN_USR below) and method calls returning a >>>> string (e.g. jdbcUrlNoData() below)? >>>> >>>> withCredentials([usernamePassword(credentialsId: >>>> 'credentialsGenerateFromDb', >>>> usernameVariable: 'GEN_USR', >>>> passwordVariable: 'GEN_PASSWD')]) { >>>> String m2RepoIdentifier = "${env.BRANCH_NAME}_${env.BUILD_NUMBER}" >>>> >>>> withEnv(javaAndMavenEnvArray()) { >>>> sh '${JAVA_HOME}/bin/java -version' // single quotes - this works >>>> fine! >>>> >>>> sh script: ''' >>>> ${mvnCommand(m2RepoIdentifier)}\ >>>> -f proj-reactor/pom.xml\ >>>> -P GenerateStorables -DdeployAtEnd=true\ >>>> -Dproj.build.generatefromdb.url=${jdbcUrlNoData()}\ >>>> -Dproj.build.generatefromdb.user=${GEN_USR}\ >>>> -Dproj.build.generatefromdb.password=${GEN_PASSWD}\ >>>> clean ${mvnBuildGoal} pmd:pmd pmd:cpd\ >>>> ''' >>>> } >>>> } >>>> >>>> On Tue, Nov 17, 2020 at 2:06 PM 'Dirk Heinrichs' via Jenkins Users < >>>> [email protected]> wrote: >>>> >>>>> Am Dienstag, den 17.11.2020, 13:45 +0100 schrieb ST: >>>>> >>>>> " '-Dproj.build.generatefromdb.user=${env.GEN_USR}'" >>>>> >>>>> >>>>> Try with ${GEN_USR} inside shell scripts, not ${env.GEN_USR}. >>>>> >>>>> HTH... >>>>> >>>>> Dirk >>>>> >>>>> -- >>>>> >>>>> *Dirk Heinrichs* >>>>> Senior Systems Engineer, Delivery Pipeline >>>>> OpenText ™ Discovery | Recommind >>>>> *Phone*: +49 2226 15966 18 <+49%202226%201596618> >>>>> *Email*: [email protected] >>>>> *Website*: www.recommind.de >>>>> Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach >>>>> Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu >>>>> Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, >>>>> Registernummer HRB 10646 >>>>> This e-mail may contain confidential and/or privileged information. If >>>>> you are not the intended recipient (or have received this e-mail in >>>>> error) >>>>> please notify the sender immediately and destroy this e-mail. Any >>>>> unauthorized copying, disclosure or distribution of the material in this >>>>> e-mail is strictly forbidden >>>>> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte >>>>> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese >>>>> E-Mail >>>>> irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und >>>>> vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte >>>>> Weitergabe dieser Mail sind nicht gestattet. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Jenkins Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/jenkinsci-users/4dd32ea372d43c9dde0800bed9a16253d45cb2e5.camel%40opentext.com >>>>> >>>>> <https://groups.google.com/d/msgid/jenkinsci-users/4dd32ea372d43c9dde0800bed9a16253d45cb2e5.camel%40opentext.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/bed31997-28f8-4e9b-b205-43f02d4348a5n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/jenkinsci-users/bed31997-28f8-4e9b-b205-43f02d4348a5n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/b5086225-d400-4500-a936-0413c025edf6n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/jenkinsci-users/b5086225-d400-4500-a936-0413c025edf6n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/69dc4fce-d19a-4cc4-b38a-b3029dca9d44n%40googlegroups.com.
