Re: Skill set to maintain a enterprise build system using Maven
For my case, I am very fortunate to involve with the the product from early day (a year ba ck) and Maven is embraced to the max where plugins are developed to solve every build use case in a full dev/qa/releng integration pipeline. The developments are multi-sites and heavily depending Maven Central like repository cluster and Maven release plugin. The odd thing to me is it is hard to find Java talent with a passion for build/ci with Maven, and it is also hard to switch perl/python devops into java to maintain Maven build. am I wrong here? Thanks -Dan On Sun, May 31, 2015 at 3:16 PM, Benson Margulies bimargul...@gmail.com wrote: First, treating build as a separate discipline from code is, in my experience, a recipe for trouble. The poms or build.xml or whatever files are just as much part of the source code as the java. Someone may own Jenkins or whatever, but the devs should own the building of the code they write. Second, the environmental impact of Maven depends critically on the nature of the code. If your entire build is composed of compiling Java code and delivering jar files, the poms are trivial and the Maven cost is near-zero. If, on the other extreme, you insist on fighting with Maven and having policies for dependencies, or releases, or whatever, that require extensive 'negotiation', it's a very different picture. In between, if you have significant custom build activity (e.g. code generators), you might need some of your own plugins. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Skill set to maintain a enterprise build system using Maven
First, treating build as a separate discipline from code is, in my experience, a recipe for trouble. The poms or build.xml or whatever files are just as much part of the source code as the java. Someone may own Jenkins or whatever, but the devs should own the building of the code they write. Second, the environmental impact of Maven depends critically on the nature of the code. If your entire build is composed of compiling Java code and delivering jar files, the poms are trivial and the Maven cost is near-zero. If, on the other extreme, you insist on fighting with Maven and having policies for dependencies, or releases, or whatever, that require extensive 'negotiation', it's a very different picture. In between, if you have significant custom build activity (e.g. code generators), you might need some of your own plugins. - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Skill set to maintain a enterprise build system using Maven
On 1 June 2015 at 09:40, Dan Tran dant...@gmail.com wrote: For my case, I am very fortunate to involve with the the product from early day (a year ba ck) and Maven is embraced to the max where plugins are developed to solve every build use case in a full dev/qa/releng integration pipeline. The developments are multi-sites and heavily depending Maven Central like repository cluster and Maven release plugin. The odd thing to me is it is hard to find Java talent with a passion for build/ci with Maven, and it is also hard to switch perl/python devops into java to maintain Maven build. am I wrong here? The odd thing to me is it is hard to find *any* talent with a passion for build/ci with *anything*. FTFY. Its a niche market, and one that lots of people still dont see the value in. Getting them to be excited about something tangential to development is difficult - its like eating healthy and excerise, people know they should be doing it but they aint. I *still* see people attempting to develop software with no version control, and hand crafted outputs from IDEs on their local machines. We even get contractors coming in wanting to use VMs in dev that they promote into production *shudder*. They best you can do is to hope you can train your people internally about why this is all good. I'm sure I've said it before, but part of Maven's problem is that this is all magically taken care of behind the scenes and less people need to know how it works to make it work. The downside is that there are then less people who can fix things when they need fixing.
Re: Skill set to maintain a enterprise build system using Maven
I'm sure I've said it before, but part of Maven's problem is that this is all magically taken care of behind the scenes and less people need to know how it works to make it work. The downside is that there are then less people who can fix things when they need fixing. Exactly, it is hard to learn magical thing I used to train a QA turned dev with java j2ee server testing backround, and it took a couple of years is it norm here by slowly training internally from start ( ie java)? Thanks -Dan BTW, i full understand that this type of discussion my be touchy, so I also very appreciated to any one who is able to share with private email
RE: Dependency plugin: tree, CLI substitution error
Hi Sander,
That was a big help! Thanks. And yes, surrounding the property in single quotes
solved the problem. For example: -Dexcludes='${project.groupId}’
FWIW, the ${groupId} form did work (so I don’t think BASH was substituting the
empty string because no such environment variable existed).
Mystery solved, not a maven issue, just my misuse of bash.
Dan
From: Sander Verhagen san...@sanderverhagen.net
mailto:san...@sanderverhagen.net
Subject: RE: Dependency plugin: tree, CLI substitution error
Date: May 30, 2015 at 2:28:11 AM EDT
To: Maven Users List users@maven.apache.org mailto:users@maven.apache.org
Hi,
Seems like your property is interpreted by BASH, which I don't think is what
you want.
${BLAH} is a BASH variable, mostly equivalent to $BASH. I think what you meant
is '${BLAH}', i.e. not substituted by BASH, giving Maven a chance to
substitute. FYI, you wouldn't want ${BLAH} either (with double quotes BASH
will still substitute).
${BLAH.BLAH} is an invalid BASH variable, due to the period, hence its balking.
Your ${groupId} probably works in the sense that BASH doesn't complain. I
don't think that it works in the sense that Maven substitutes it for the actual
groupId. I don't think that it'll get the chance for that, because likely BASH
already substitutes it for an empty string (assuming it doesn't know a groupId
variable in your environment; not entirely sure on basis of the limited
information given).
Hope this helps.
Sander Verhagen
[ san...@sanderverhagen.net mailto:san...@sanderverhagen.net ]
Re: Skill set to maintain a enterprise build system using Maven
Hello Dan, we treat tooling like software as well. Ticket creation is an automated 2 click process and the package qa will not take more than 5 minutes for small changes. External libraries from central may be used at free will, but we recommend stuff in a so called toolbox, these dependencies are managed in the department pom. The (programming) architects and we help discovering alternatives in our toolbox, stuff from repositories outside central is mostly put in a third-party repo in Artifactory. Regards Mirko -- Sent from my mobile Am 31.05.2015 00:06 schrieb Dan Tran dant...@gmail.com: Hi Mirko Looks like you have Artifactory to store all of release artifacts and another 'release' repo to store the final approved release Is internal tooling, thirdparty upload going thru the same release process? Thanks -Dan On Sat, May 30, 2015 at 2:41 PM, Mirko Friedenhagen mfriedenha...@gmail.com wrote: Hello Dan, - Every developer may deploy SNAPSHOTs, however this is normally done by Jenkins. - We do not enforce staging from Jenkins, however almost all projects do this. We do not enforce this, so Jenkins outages do not inhibit releasing hot fixes. - Releases are deployed to a staging repository in Artifactory and we have a process called package-qa where for every staged release a corresponding JIRA ticket has to be created with information (changes, Wiki page, diff link since last release, ticket queue). This is a central place where you may see all releases in one place. - This ticket is parsed by a script from Jenkins which procures artifacts from staging to a releases repository and adds general quality information from SonarQube and Jenkins as well as the SHA1 sums to the ticket so we have a second record which may be used to detect forgery. Additionally the script checks for the existence of the SCM tag and retrieves the number of changed lines between releases. - No blocker or critical and no new major issues are allowed in SonarQube, otherwise procurement will fail. - The reporter and the mover have to be different persons to enforce a four eyes principle. - The mover (sometimes someone from development QA, most of the times nowadays another developer) has to check some things and must inspect the diff to detect whether all changes are explained. - Our operations teams will only pick production releases from the final releases repository, other stages may pick up artifacts from the staging repository. - We do not sign artifacts. Regards Mirko Regards Mirko -- http://illegalstateexception.blogspot.com/ https://github.com/mfriedenhagen/ (http://osrc.dfm.io/mfriedenhagen) https://bitbucket.org/mfriedenhagen/ On Sat, May 30, 2015 at 7:31 PM, Dan Tran dant...@gmail.com wrote: Thanks Mirko * What about snapshot and release policy, do developers/qa have access to deploy snapshot and release artifacts? * do you use artifact signing similar to Maven Central? Thanks again -Dan On Sat, May 30, 2015 at 9:21 AM, Mirko Friedenhagen mfriedenha...@gmail.com wrote: What I forgot: patience, social skills and remembering that not every application developer needs to be a build specialist are important as well :-) Regards Mirko -- Sent from my mobile Am 30.05.2015 07:29 schrieb Dan Tran dant...@gmail.com: Hi I would like to ask if the community can share with me what it takes to maintain an enterprise build system with continuous integration of 100+ developer + QA and growing using Maven. The build system contains many components with their own release cycle and they do integrate together. - is java skill set to develop plugin a must? - do you have a team or just a few of deep understanding of Maven developers? - will a none java RelEng able to perform Maven release? - does your RelEng maintains the pom or developers? - what are your challenges? Thanks -Dan - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Skill set to maintain a enterprise build system using Maven
Hi Curtis, It is an awesome info and process, I have a long way to catch up Thanks -Dan On Sun, May 31, 2015 at 2:36 AM, Curtis Rueden ctrue...@wisc.edu wrote: Hi Dan, PS. Would love to hear other experiences from community rather me sucking out Mirko's :-) Not sure how relevant my scenario is, but here goes: My group consists of an international collaboration of OSS developers at universities etc., rather than a company. But a lot of our needs are the same. - We have 2-3 core DevOps who maintain our Maven architecture (corporate POM [1] etc.), with several others in the community who have been trained enough on how things work to understand and occasionally contribute to it. - The projects encompass several GitHub orgs, each of which has its own primary maintainer(s) who at least understand how their Maven POM hierarchy works. - We have Jenkins jobs for everything. http://jenkins.imagej.net/ - We use Nexus to manage our Maven artifacts. http://maven.imagej.net/ - is java skill set to develop plugin a must? No, although it is occasionally useful. I think knowing Java can make you a better Maven community member though because you can contribute bug-fixes to the plugins you find useful. E.g., we use license-maven-plugin, but it wouldn't have behaved the way we needed if I hadn't helped it get patched upstream. - do you have a team or just a few of deep understanding of Maven developers? 2-3 with deep understanding and several others worldwide with moderate understanding has been sufficient for us. And a whole community with next to no understanding of Maven, for whom things just work in their IDE of choice. - will a none java RelEng able to perform Maven release? I don't see why not, although in my scenario everyone who knows about Maven also knows about Java. So for us it is the converse: lots of Java developers who know little about Maven but want to do their own releases. And the beauty of it is: they can! http://imagej.net/Releases - does your RelEng maintains the pom or developers? My group has no dedicated RelEng—just two core developers who also wear the RelEng hat among many others. In practice, for the most part, the same people who ultimately maintain the most root-level POMs also end up doing the releases of the most root-level software projects. - what are your challenges? Maven makes it possible for a small number of developers to maintain 300+ Git repositories. This is a two edged sword, though: Maven only solves the project management side of things, not the other aspects (software design/architecture, coding of new features, community support, etc.). It is important to note that there are many project management challenges that Maven makes easier to solve but does not itself fully solve in a turn-key way. In particular, continuous integration (thank you Jenkins!) and versioning (we use SemVer and carefully managed BOMs) spring to mind. Since everything we do is OSS, if you care you can read about it yourself: http://imagej.net/Architecture http://imagej.net/Governance Regards, Curtis On Sun, May 31, 2015 at 4:06 AM, Dan Tran dant...@gmail.com wrote: Hi Mirko, Looks like the topic getting very interesting here, but I would like to refocus on the skillset How did you form up your team?, did you all pickup Java before joining this devops team? Thanks for all the sharing -Dan PS. Would love to hear other experiences from community rather me sucking out Mirko's :-) On Sun, May 31, 2015 at 1:03 AM, Mirko Friedenhagen mfriedenha...@gmail.com wrote: Hello Dan, we treat tooling like software as well. Ticket creation is an automated 2 click process and the package qa will not take more than 5 minutes for small changes. External libraries from central may be used at free will, but we recommend stuff in a so called toolbox, these dependencies are managed in the department pom. The (programming) architects and we help discovering alternatives in our toolbox, stuff from repositories outside central is mostly put in a third-party repo in Artifactory. Regards Mirko -- Sent from my mobile Am 31.05.2015 00:06 schrieb Dan Tran dant...@gmail.com: Hi Mirko Looks like you have Artifactory to store all of release artifacts and another 'release' repo to store the final approved release Is internal tooling, thirdparty upload going thru the same release process? Thanks -Dan On Sat, May 30, 2015 at 2:41 PM, Mirko Friedenhagen mfriedenha...@gmail.com wrote: Hello Dan, - Every developer may deploy SNAPSHOTs, however this is normally done by Jenkins. - We do not enforce staging from Jenkins, however almost all projects do this. We do not enforce this, so Jenkins outages do not inhibit releasing hot fixes. -
Re: Skill set to maintain a enterprise build system using Maven
Hi Mirko, Looks like the topic getting very interesting here, but I would like to refocus on the skillset How did you form up your team?, did you all pickup Java before joining this devops team? Thanks for all the sharing -Dan PS. Would love to hear other experiences from community rather me sucking out Mirko's :-) On Sun, May 31, 2015 at 1:03 AM, Mirko Friedenhagen mfriedenha...@gmail.com wrote: Hello Dan, we treat tooling like software as well. Ticket creation is an automated 2 click process and the package qa will not take more than 5 minutes for small changes. External libraries from central may be used at free will, but we recommend stuff in a so called toolbox, these dependencies are managed in the department pom. The (programming) architects and we help discovering alternatives in our toolbox, stuff from repositories outside central is mostly put in a third-party repo in Artifactory. Regards Mirko -- Sent from my mobile Am 31.05.2015 00:06 schrieb Dan Tran dant...@gmail.com: Hi Mirko Looks like you have Artifactory to store all of release artifacts and another 'release' repo to store the final approved release Is internal tooling, thirdparty upload going thru the same release process? Thanks -Dan On Sat, May 30, 2015 at 2:41 PM, Mirko Friedenhagen mfriedenha...@gmail.com wrote: Hello Dan, - Every developer may deploy SNAPSHOTs, however this is normally done by Jenkins. - We do not enforce staging from Jenkins, however almost all projects do this. We do not enforce this, so Jenkins outages do not inhibit releasing hot fixes. - Releases are deployed to a staging repository in Artifactory and we have a process called package-qa where for every staged release a corresponding JIRA ticket has to be created with information (changes, Wiki page, diff link since last release, ticket queue). This is a central place where you may see all releases in one place. - This ticket is parsed by a script from Jenkins which procures artifacts from staging to a releases repository and adds general quality information from SonarQube and Jenkins as well as the SHA1 sums to the ticket so we have a second record which may be used to detect forgery. Additionally the script checks for the existence of the SCM tag and retrieves the number of changed lines between releases. - No blocker or critical and no new major issues are allowed in SonarQube, otherwise procurement will fail. - The reporter and the mover have to be different persons to enforce a four eyes principle. - The mover (sometimes someone from development QA, most of the times nowadays another developer) has to check some things and must inspect the diff to detect whether all changes are explained. - Our operations teams will only pick production releases from the final releases repository, other stages may pick up artifacts from the staging repository. - We do not sign artifacts. Regards Mirko Regards Mirko -- http://illegalstateexception.blogspot.com/ https://github.com/mfriedenhagen/ (http://osrc.dfm.io/mfriedenhagen) https://bitbucket.org/mfriedenhagen/ On Sat, May 30, 2015 at 7:31 PM, Dan Tran dant...@gmail.com wrote: Thanks Mirko * What about snapshot and release policy, do developers/qa have access to deploy snapshot and release artifacts? * do you use artifact signing similar to Maven Central? Thanks again -Dan On Sat, May 30, 2015 at 9:21 AM, Mirko Friedenhagen mfriedenha...@gmail.com wrote: What I forgot: patience, social skills and remembering that not every application developer needs to be a build specialist are important as well :-) Regards Mirko -- Sent from my mobile Am 30.05.2015 07:29 schrieb Dan Tran dant...@gmail.com: Hi I would like to ask if the community can share with me what it takes to maintain an enterprise build system with continuous integration of 100+ developer + QA and growing using Maven. The build system contains many components with their own release cycle and they do integrate together. - is java skill set to develop plugin a must? - do you have a team or just a few of deep understanding of Maven developers? - will a none java RelEng able to perform Maven release? - does your RelEng maintains the pom or developers? - what are your challenges? Thanks -Dan - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
Re: Skill set to maintain a enterprise build system using Maven
Hi Dan, PS. Would love to hear other experiences from community rather me sucking out Mirko's :-) Not sure how relevant my scenario is, but here goes: My group consists of an international collaboration of OSS developers at universities etc., rather than a company. But a lot of our needs are the same. - We have 2-3 core DevOps who maintain our Maven architecture (corporate POM [1] etc.), with several others in the community who have been trained enough on how things work to understand and occasionally contribute to it. - The projects encompass several GitHub orgs, each of which has its own primary maintainer(s) who at least understand how their Maven POM hierarchy works. - We have Jenkins jobs for everything. http://jenkins.imagej.net/ - We use Nexus to manage our Maven artifacts. http://maven.imagej.net/ - is java skill set to develop plugin a must? No, although it is occasionally useful. I think knowing Java can make you a better Maven community member though because you can contribute bug-fixes to the plugins you find useful. E.g., we use license-maven-plugin, but it wouldn't have behaved the way we needed if I hadn't helped it get patched upstream. - do you have a team or just a few of deep understanding of Maven developers? 2-3 with deep understanding and several others worldwide with moderate understanding has been sufficient for us. And a whole community with next to no understanding of Maven, for whom things just work in their IDE of choice. - will a none java RelEng able to perform Maven release? I don't see why not, although in my scenario everyone who knows about Maven also knows about Java. So for us it is the converse: lots of Java developers who know little about Maven but want to do their own releases. And the beauty of it is: they can! http://imagej.net/Releases - does your RelEng maintains the pom or developers? My group has no dedicated RelEng—just two core developers who also wear the RelEng hat among many others. In practice, for the most part, the same people who ultimately maintain the most root-level POMs also end up doing the releases of the most root-level software projects. - what are your challenges? Maven makes it possible for a small number of developers to maintain 300+ Git repositories. This is a two edged sword, though: Maven only solves the project management side of things, not the other aspects (software design/architecture, coding of new features, community support, etc.). It is important to note that there are many project management challenges that Maven makes easier to solve but does not itself fully solve in a turn-key way. In particular, continuous integration (thank you Jenkins!) and versioning (we use SemVer and carefully managed BOMs) spring to mind. Since everything we do is OSS, if you care you can read about it yourself: http://imagej.net/Architecture http://imagej.net/Governance Regards, Curtis On Sun, May 31, 2015 at 4:06 AM, Dan Tran dant...@gmail.com wrote: Hi Mirko, Looks like the topic getting very interesting here, but I would like to refocus on the skillset How did you form up your team?, did you all pickup Java before joining this devops team? Thanks for all the sharing -Dan PS. Would love to hear other experiences from community rather me sucking out Mirko's :-) On Sun, May 31, 2015 at 1:03 AM, Mirko Friedenhagen mfriedenha...@gmail.com wrote: Hello Dan, we treat tooling like software as well. Ticket creation is an automated 2 click process and the package qa will not take more than 5 minutes for small changes. External libraries from central may be used at free will, but we recommend stuff in a so called toolbox, these dependencies are managed in the department pom. The (programming) architects and we help discovering alternatives in our toolbox, stuff from repositories outside central is mostly put in a third-party repo in Artifactory. Regards Mirko -- Sent from my mobile Am 31.05.2015 00:06 schrieb Dan Tran dant...@gmail.com: Hi Mirko Looks like you have Artifactory to store all of release artifacts and another 'release' repo to store the final approved release Is internal tooling, thirdparty upload going thru the same release process? Thanks -Dan On Sat, May 30, 2015 at 2:41 PM, Mirko Friedenhagen mfriedenha...@gmail.com wrote: Hello Dan, - Every developer may deploy SNAPSHOTs, however this is normally done by Jenkins. - We do not enforce staging from Jenkins, however almost all projects do this. We do not enforce this, so Jenkins outages do not inhibit releasing hot fixes. - Releases are deployed to a staging repository in Artifactory and we have a process called package-qa where for every staged release a corresponding JIRA ticket has to be created with information (changes, Wiki page, diff link since last release, ticket queue). This is a central
Re: Skill set to maintain a enterprise build system using Maven
On 30/05/2015 10:44 PM, Dan Tran wrote: Hi Ron One person may not be desirable since he/she may win a lottery :-) I sell Learning Management Systems that include Talent and Succession Planning so I will just say that you should be able to find a backup in your pool of potential successors for any key position.:-) The actual change to a POM is not a high skill job once you have the project's policies and initial POMs in place. Developers should not change the POM who would? RelEng? It should be set up by a senior architect or architectural team that is making and supporting the team's policies for dependency selection and release strategy. We had pretty good success with simple POMs that individual developers were not permitted (not much enforcement required) to alter. If a developer has a need for a new dependency or a change in version, it has to be a team decision with the project manager's OK. Ron Thanks -Dan On Sat, May 30, 2015 at 3:22 PM, Ron Wheeler rwhee...@artifact-software.com wrote: On 30/05/2015 1:29 AM, Dan Tran wrote: Hi I would like to ask if the community can share with me what it takes to maintain an enterprise build system with continuous integration of 100+ developer + QA and growing using Maven. The build system contains many components with their own release cycle and they do integrate together. - is java skill set to develop plugin a must? a) Most projects should not need to develop plug-ins. b) This is a one-time need if it exists and there are companies and consultants available to do this for you - do you have a team or just a few of deep understanding of Maven developers? One personwill do. - will a none java RelEng able to perform Maven release? Sure - does your RelEng maintains the pom or developers? developers should not touch POMs - what are your challenges? a) adjusting your project structure and methodology to incorporate the maven way b) resisting the impulse to try to impose an existing mindset about development and build processes on Maven Thanks -Dan -- Ron Wheeler President Artifact Software Inc email: rwhee...@artifact-software.com skype: ronaldmwheeler phone: 866-970-2435, ext 102 - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org -- Ron Wheeler President Artifact Software Inc email: rwhee...@artifact-software.com skype: ronaldmwheeler phone: 866-970-2435, ext 102 - To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
