Sorry, I'm a bit late to the party on this one :). +1 on the four builds Nick described. Each would be useful and purpose built.
I like the idea of using Docker, especially for local development and quick testing. Has anyone explored this? I'm envisioning very specific containers so you could spin up only the components you're actively working on. Certainly something I would be willing to put some cycles into if there is interest. -Kyle On Fri, Oct 14, 2016 at 2:15 PM, Ryan Merriman <[email protected]> wrote: > +1 I like it. Just to clarify, the scripts to run Storm topologies locally > in an IDE should be available independent of the environment running. No > need for a separate build/image. > > On Fri, Oct 14, 2016 at 9:12 AM, Otto Fowler <[email protected]> > wrote: > > > Going forward, the Demo env and data would have implications for testing > as > > well ( gold data sets ) etc. > > > > On October 14, 2016 at 09:52:07, Nick Allen ([email protected]) wrote: > > > > I think based on everyone's input so far, we're describing 4 different > > builds/images/tools that would each be intended to run on a standard > > Mac/Linux/Windows laptop. > > > > Full Dev - A development environment that performs a full end-to-end > > deployment of Metron. This is intended for developers working with > > sensors, deployments, or validating how all Metron components interact > with > > one another. > > > > > > - Starts from base Linux image > > - Installs Hadoop-y components > > - Installs Metron > > - Installs Sensors > > - Nothing started by default > > > > Quick Dev - An environment intended for the developer focusing on the > > streaming components of Metron; parsing, enrichment, and indexing. > > > > > > - Starts from base image of Linux + Hadoop-y components > > - Installs Metron > > - Installs "data generator" spouts > > - Does not install sensors > > - Nothing started by default > > > > Demo - An environment intended to introduce new users to Metron. The > > environment should go from nothing to plenty of data in the Metron > > dashboard in as little "boot" time as possible. > > > > > > - Starts from a base image including Linux + Hadoop-y + Metron + Data > > Generator Spouts pre-installed > > - Pre-load Elasticsearch indices so the user has plenty of data to > > investigate in the dashboard > > - Does not install sensors > > - Everything started by default > > > > Storm Local Cluster - Otto suggested some scripts/tooling to make it easy > > to launch the core topologies on a local Storm cluster running on the > host > > OS. > > > > > > I'd be interested to hear if this works for everyone and how this might > > play into the Ambari mpack + RPM based deployment scheme. > > > > > > On Fri, Oct 14, 2016 at 1:45 AM, Michael Miklavcic < > > [email protected]> wrote: > > > > > I think this may have come up in another PR already (have to look for > > it). > > > But maybe we could maintain our flexibility in quick-dev by installing > > the > > > sensors and not starting them until we need them. I think it's useful > to > > > have a quick "genuine" e2e testing environment that doesn't require > > running > > > through a full install. I'm also not opposed to extracting the > > integration > > > test functionality into general purpose data generators. > > > > > > On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen <[email protected]> > wrote: > > > > > > > To Jon's point, I think it would be useful to have a Demo box that > uses > > > > generators to produce 3 or 4 types of telemetry that shows up in the > > > Metron > > > > Dashboard. This box would be different from Quick-Dev in that > > everything > > > > starts automatically, so that a user just has to launch it and the > > should > > > > start seeing data in the Metron Dashboard right away. In fact, we > could > > > > even pre-load the Elasticsearch indices so that the user has more of > a > > > > history to mine when using the Demo box. > > > > > > > > On Thu, Oct 13, 2016 at 2:04 PM, [email protected] <[email protected]> > > > > wrote: > > > > > > > > > +1 Ryan and Otto's comments. > > > > > > > > > > I also strongly think we need to make a demo environment easier, > but > > > that > > > > > should be different than quick-dev. > > > > > > > > > > Jon > > > > > > > > > > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > - create scripts/utilities to easily run a topology locally in an > > IDE > > > > > > instead of in the VM > > > > > > > > > > > > > > > > > > ^^^^^^^^ THIS. > > > > > > > > > > > > > > > > > > On October 13, 2016 at 12:36:45, Ryan Merriman ( > > [email protected]) > > > > > > > > wrote: > > > > > > > > > > > > Working with the quick-dev vagrant VM recently left a lot to be > > > > desired. > > > > > > All forthcoming comments are made under the assumption that this > VM > > > is > > > > > > intended for development purposes. If that is not true, I think > we > > > > should > > > > > > consider adding a VM for this purpose (or Docker containers?). > Here > > > are > > > > > > the issues I ran into that I think can be improved: > > > > > > > > > > > > - had to upgrade VirtualBox from 5.0.16 to 5.0.20 > > > > > > - had to update to the latest metron/hdp-base Vagrant box > > > > > > - takes forever to spin up > > > > > > - VM is constrained for resources making it unstable > > > > > > - spent a large amount of time troubleshooting sensors (no raw > > > messages > > > > > > in Kafka) > > > > > > - no easy way to debug topologies > > > > > > > > > > > > Fortunately I think we can make this a much better experience > > > without a > > > > > > major effort. Here are my ideas to do this: > > > > > > > > > > > > - update the prereqs for VirtualBox > > > > > > - add a check for the appropriate base box version (Jira has > > already > > > > > > been created https://issues.apache.org/jira/browse/METRON-497) > > > > > > - don't install any sensors and replace them with a data > generator > > > that > > > > > > just loops through sample data and emits to Kafka (could also be > > used > > > > to > > > > > > replay and troubleshoot edge cases) > > > > > > - everything in monit is off by default except for ES or other > > > critical > > > > > > services > > > > > > - create scripts/utilities to easily run a topology locally in an > > IDE > > > > > > instead of in the VM > > > > > > - improved documentation with examples of how to run and > > troubleshoot > > > > > > topologies > > > > > > > > > > > > Is this a worthwhile effort? I think this would also give users > an > > > > easier > > > > > > path to demonstrate or tour Metron's capabilities. Are there any > > > other > > > > > > improvements people would like to see? Should we wait for Docker? > > > > > > Thoughts? > > > > > > > > > > > > Ryan Merriman > > > > > > > > > > > -- > > > > > > > > > > Jon > > > > > > > > > > > > > > > > > > > > > -- > > > > Nick Allen <[email protected]> > > > > > > > > > > > > > > > -- > > Nick Allen <[email protected]> > > >
