Matt, if you want it to be automated you will need to use our integration testing framework since Docker isn't a part of our build process right now. If you want to use Docker to aid in your development and get you to a point where things are working as expected, I think it would work well for that.
Ryan On Mon, Feb 6, 2017 at 12:29 PM, Matt Foley <[email protected]> wrote: > There may be another area of application for this. I’m not certain, so > tell me if I’m off base. > > In the context of METRON-322 (adding batchTimeout and > 'topology.tick.tuple.freq.secs' to BulkMessageWriterBolt), there are some > fairly obvious unit tests needed, but I find them inadequate to give me > certainty that some fairly complex-interaction changes are actually doing > what they’re supposed to be doing. So I was thinking how to do integration > testing of only a Bolt component (or worst case, only an Indexing topology) > outside of spinning up a whole Metron / Hadoop Stack cluster. I wasn’t > coming up with any good answers :-) so I was about to ask the list anyway. > > Is it feasible/advisable to use Docker to do automated integration testing > of small chunks of Metron, such as a single component or a single > topology? What’s doable? Other better ideas? > > Thanks, > --Matt > > > On 2/6/17, 10:03 AM, "Ryan Merriman" <[email protected]> wrote: > > From the README: > > "Metron Docker is a Docker Compose application that is intended for > development and integration testing of Metron. Use this instead of > Vagrant > when: > > - You want an environment that can be built and spun up quickly > - You need to frequently rebuild and restart services > - You only need to test, troubleshoot or develop against a subset of > services" > > The "Quick Dev" environment actually serves 2 purposes: a development > environment and an end-to-end testing environment. This module was > intended to supplement or provide an alternative to the development > environment part of "Quick Dev", not the end-to-end testing part. It > does > have "Docker" in the name of the module so I can see how that might > suggest > a fully supported deployment option. It shouldn't be used for that > though > because it doesn't include Ambari or MPack and isn't a true > representation > of a production Metron cluster. > > What is the direction? I could see this evolving into a collection of > profiles or recipes. Need to development a custom parser? Spin up an > application that only includes the Storm, Kafka and Zookeeper images. > Want > to develop a custom Kibana dashboard? Spin up Elasticsearch and Kibana > images preloaded with data. Maybe an analytics profile could be > created > that only includes the tools you need for that? The application that > exists now in metron-docker could be considered a "rest" profile or a > collection of containers that support all the functions of the rest > API. > It's very general purpose and supports a lot of use cases so I > considered > it a good starting point. It's very useful if you're developing a UI > and > have limited knowledge of Ambari or big data platform services. That > was > the initial motivation. > > I think you should view this as more of a toolbox and not a turnkey > installation solution. Maintaining and building development > environments > is something Docker is a really good fit for and I have found this > works > much better than our Ansible/Vagrant environment. It's really fast and > stays up all the time. > > But it's completely optional. Use it if you think it will help you. > Or > don't if "Quick Dev" is good enough and you've figured out how to tune > it > so that it's not completely unusable. If everybody thinks it's > confusing > and no one uses it then we can take it out and I'll just go back to > maintaining it privately. But then I would miss out on Kyle's awesome > contribution :) > > Ryan > > On Mon, Feb 6, 2017 at 10:12 AM, Nick Allen <[email protected]> > wrote: > > > So what is the direction then, Ryan? Can you describe what this is > > supposed to be used for? > > > > I had thought people wanted this to replace the existing > Vagrant-based > > "Quick Dev"? But apparently this is the assumption that you think I > am > > wrong on. > > > > > > > > On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <[email protected]> > > wrote: > > > > > I agree with everything Kyle said and I think some of Nick's > assumptions > > > are false. I don't see this a third deployment option. > > > > > > I can understand people not wanting to maintain another deployment > path > > > with Metron already being as big as it is. Ensuring that you've > tested > > and > > > updated all the appropriate components is already tedious. But in > the > > case > > > of this module, is it something that needs to updated anytime > someone > > makes > > > a deployment related change? I don't think so and I've never had > that > > > expectation. The build won't fail and nothing from this project > is ever > > > deployed or shipped. For me, maintaining this tool as needed is > good > > > enough. What happens if a change is introduced that breaks > something? I > > > discover it as I'm using the tool, fix it, contribute it back and > move > > on. > > > No big deal. I had been maintaining this privately for a while > before > > the > > > PR was submitted and the work to keep it current with master was > pretty > > > minimal. Does that mean it should live somewhere else besides the > master > > > branch in Metron? I'm not sure what the answer is but there > should be a > > > way to share and collaborate with the community on tools like this > that > > > aren't necessarily deployed to production. Kyle's contribution is > > valuable > > > and something I would definitely use. > > > > > > Ryan > > > > > > > > >
