So we use Bamboo for our CI. Basically a commit comes in, Bamboo picks up the change and runs our automated test suite. If the tests all pass, we have bamboo package up a jar. As part of this process we write the version information and bamboo build number into a properties file in the jar. Bamboo then publishes the jar to Artifactory.
As of now, when we want to redeploy a topology we have a simple shell script (manual process, tho it could be automated if we wanted) that polls Artifactory for the available builds that can be deployed, you select which one you want to ship, and it handles deploying it out to our cluster. When the topology is deployed, it reads the version properties file and writes that into the topologies config. This way we can inspect a running topology via the webui and easily see what version/build is currently out in production. Additionally our deployment script records when and what builds are deployed. I'd love to hear what other people are doing! On Wed, Jan 6, 2016 at 12:55 AM, Tom Raney <[email protected]> wrote: > Hi Zack, > > I'm also interested in hearing more about how folks generally handle the > deployment process. In particular, versioning the topology in such a way > that makes continuous integration possible without manual deployment steps. > > This project (https://github.com/ptgoetz/storm-maven-plugin) seems like a > step forward to integrating with Jenkins nicely, but it appears abandoned > or perhaps replaced by something else? I'm not against using the Maven > exec plugin to build something, but, I'd hate to do that when someone else > has already built a solution. > > As far as handling the repo itself, I think it depends on how many spouts > and bolts you're planning on re-using across different topologies. And, > unless the upload time becomes burdensome, it doesn't seem that terrible to > just maintain a single large super jar. If the jar size is an issue, you > could break your project into Maven sub-modules within a single repo and > have custom shaded builders for particular topologies to reduce the jar > size. > > Regards, > > Tom > > On Mon, Jan 4, 2016 at 3:50 PM, Zack Allen <[email protected]> wrote: > >> Hi all, >> >> I would like to know the best practices in terms of the submission and >> management of storm topologies. >> >> Do you manage 1 repo per topology, or have 1 repo with many topologies in >> it? For example, I have a topology that counts words, a topology that sees >> if an input is a prime number, and a topology that adds exclamation points >> to a piece of text. >> >> What would be the best option in terms of updating the prime number >> topology? I know you can remove one from the UI, but do you generally ship >> a .jar with all of the topologies or do you have one strictly for that >> piece of processing? I am leaning towards 1 repo for 1 topology to make >> deployments more intuitive (push a change via github, assemble the .jar, >> push to storm topology) but I would like to hear what everyone here does. >> >> Thanks! >> > >
