+1 I like it.  Just to clarify, the scripts to run Storm topologies locally
in an IDE should be available independent of the environment running.  No
need for a separate build/image.

On Fri, Oct 14, 2016 at 9:12 AM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Going forward, the Demo env and data would have implications for testing as
> well ( gold data sets ) etc.
>
> On October 14, 2016 at 09:52:07, Nick Allen (n...@nickallen.org) wrote:
>
> I think based on everyone's input so far, we're describing 4 different
> builds/images/tools that would each be intended to run on a standard
> Mac/Linux/Windows laptop.
>
> Full Dev - A development environment that performs a full end-to-end
> deployment of Metron. This is intended for developers working with
> sensors, deployments, or validating how all Metron components interact with
> one another.
>
>
> - Starts from base Linux image
> - Installs Hadoop-y components
> - Installs Metron
> - Installs Sensors
> - Nothing started by default
>
> Quick Dev - An environment intended for the developer focusing on the
> streaming components of Metron; parsing, enrichment, and indexing.
>
>
> - Starts from base image of Linux + Hadoop-y components
> - Installs Metron
> - Installs "data generator" spouts
> - Does not install sensors
> - Nothing started by default
>
> Demo - An environment intended to introduce new users to Metron. The
> environment should go from nothing to plenty of data in the Metron
> dashboard in as little "boot" time as possible.
>
>
> - Starts from a base image including Linux + Hadoop-y + Metron + Data
> Generator Spouts pre-installed
> - Pre-load Elasticsearch indices so the user has plenty of data to
> investigate in the dashboard
> - Does not install sensors
> - Everything started by default
>
> Storm Local Cluster - Otto suggested some scripts/tooling to make it easy
> to launch the core topologies on a local Storm cluster running on the host
> OS.
>
>
> I'd be interested to hear if this works for everyone and how this might
> play into the Ambari mpack + RPM based deployment scheme.
>
>
> On Fri, Oct 14, 2016 at 1:45 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I think this may have come up in another PR already (have to look for
> it).
> > But maybe we could maintain our flexibility in quick-dev by installing
> the
> > sensors and not starting them until we need them. I think it's useful to
> > have a quick "genuine" e2e testing environment that doesn't require
> running
> > through a full install. I'm also not opposed to extracting the
> integration
> > test functionality into general purpose data generators.
> >
> > On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen <n...@nickallen.org> wrote:
> >
> > > To Jon's point, I think it would be useful to have a Demo box that uses
> > > generators to produce 3 or 4 types of telemetry that shows up in the
> > Metron
> > > Dashboard. This box would be different from Quick-Dev in that
> everything
> > > starts automatically, so that a user just has to launch it and the
> should
> > > start seeing data in the Metron Dashboard right away. In fact, we could
> > > even pre-load the Elasticsearch indices so that the user has more of a
> > > history to mine when using the Demo box.
> > >
> > > On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com <zeo...@gmail.com>
> > > wrote:
> > >
> > > > +1 Ryan and Otto's comments.
> > > >
> > > > I also strongly think we need to make a demo environment easier, but
> > that
> > > > should be different than quick-dev.
> > > >
> > > > Jon
> > > >
> > > > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler <ottobackwa...@gmail.com
> >
> > > > wrote:
> > > >
> > > > > - create scripts/utilities to easily run a topology locally in an
> IDE
> > > > > instead of in the VM
> > > > >
> > > > >
> > > > > ^^^^^^^^ THIS.
> > > > >
> > > > >
> > > > > On October 13, 2016 at 12:36:45, Ryan Merriman (
> merrim...@gmail.com)
>
> > > > > wrote:
> > > > >
> > > > > Working with the quick-dev vagrant VM recently left a lot to be
> > > desired.
> > > > > All forthcoming comments are made under the assumption that this VM
> > is
> > > > > intended for development purposes. If that is not true, I think we
> > > should
> > > > > consider adding a VM for this purpose (or Docker containers?). Here
> > are
> > > > > the issues I ran into that I think can be improved:
> > > > >
> > > > > - had to upgrade VirtualBox from 5.0.16 to 5.0.20
> > > > > - had to update to the latest metron/hdp-base Vagrant box
> > > > > - takes forever to spin up
> > > > > - VM is constrained for resources making it unstable
> > > > > - spent a large amount of time troubleshooting sensors (no raw
> > messages
> > > > > in Kafka)
> > > > > - no easy way to debug topologies
> > > > >
> > > > > Fortunately I think we can make this a much better experience
> > without a
> > > > > major effort. Here are my ideas to do this:
> > > > >
> > > > > - update the prereqs for VirtualBox
> > > > > - add a check for the appropriate base box version (Jira has
> already
> > > > > been created https://issues.apache.org/jira/browse/METRON-497)
> > > > > - don't install any sensors and replace them with a data generator
> > that
> > > > > just loops through sample data and emits to Kafka (could also be
> used
> > > to
> > > > > replay and troubleshoot edge cases)
> > > > > - everything in monit is off by default except for ES or other
> > critical
> > > > > services
> > > > > - create scripts/utilities to easily run a topology locally in an
> IDE
> > > > > instead of in the VM
> > > > > - improved documentation with examples of how to run and
> troubleshoot
> > > > > topologies
> > > > >
> > > > > Is this a worthwhile effort? I think this would also give users an
> > > easier
> > > > > path to demonstrate or tour Metron's capabilities. Are there any
> > other
> > > > > improvements people would like to see? Should we wait for Docker?
> > > > > Thoughts?
> > > > >
> > > > > Ryan Merriman
> > > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > >
> > >
> > >
> > > --
> > > Nick Allen <n...@nickallen.org>
> > >
> >
>
>
>
> --
> Nick Allen <n...@nickallen.org>
>

Reply via email to