Re: [DISCUSS] Improving quick-dev

[Jimson K James]

Really looking forward to dockerized Metron.
Thanks ahead.

On Wednesday, 19 October 2016, Kyle Richardson 
wrote:

> Sorry, I'm a bit late to the party on this one :).
>
> +1 on the four builds Nick described. Each would be useful and purpose
> built.
>
> I like the idea of using Docker, especially for local development and quick
> testing. Has anyone explored this? I'm envisioning very specific containers
> so you could spin up only the components you're actively working on.
> Certainly something I would be willing to put some cycles into if there is
> interest.
>
> -Kyle
>
> On Fri, Oct 14, 2016 at 2:15 PM, Ryan Merriman  > wrote:
>
> > +1 I like it.  Just to clarify, the scripts to run Storm topologies
> locally
> > in an IDE should be available independent of the environment running.  No
> > need for a separate build/image.
> >
> > On Fri, Oct 14, 2016 at 9:12 AM, Otto Fowler  >
> > wrote:
> >
> > > Going forward, the Demo env and data would have implications for
> testing
> > as
> > > well ( gold data sets ) etc.
> > >
> > > On October 14, 2016 at 09:52:07, Nick Allen (n...@nickallen.org
> ) wrote:
> > >
> > > I think based on everyone's input so far, we're describing 4 different
> > > builds/images/tools that would each be intended to run on a standard
> > > Mac/Linux/Windows laptop.
> > >
> > > Full Dev - A development environment that performs a full end-to-end
> > > deployment of Metron. This is intended for developers working with
> > > sensors, deployments, or validating how all Metron components interact
> > with
> > > one another.
> > >
> > >
> > > - Starts from base Linux image
> > > - Installs Hadoop-y components
> > > - Installs Metron
> > > - Installs Sensors
> > > - Nothing started by default
> > >
> > > Quick Dev - An environment intended for the developer focusing on the
> > > streaming components of Metron; parsing, enrichment, and indexing.
> > >
> > >
> > > - Starts from base image of Linux + Hadoop-y components
> > > - Installs Metron
> > > - Installs "data generator" spouts
> > > - Does not install sensors
> > > - Nothing started by default
> > >
> > > Demo - An environment intended to introduce new users to Metron. The
> > > environment should go from nothing to plenty of data in the Metron
> > > dashboard in as little "boot" time as possible.
> > >
> > >
> > > - Starts from a base image including Linux + Hadoop-y + Metron + Data
> > > Generator Spouts pre-installed
> > > - Pre-load Elasticsearch indices so the user has plenty of data to
> > > investigate in the dashboard
> > > - Does not install sensors
> > > - Everything started by default
> > >
> > > Storm Local Cluster - Otto suggested some scripts/tooling to make it
> easy
> > > to launch the core topologies on a local Storm cluster running on the
> > host
> > > OS.
> > >
> > >
> > > I'd be interested to hear if this works for everyone and how this might
> > > play into the Ambari mpack + RPM based deployment scheme.
> > >
> > >
> > > On Fri, Oct 14, 2016 at 1:45 AM, Michael Miklavcic <
> > > michael.miklav...@gmail.com > wrote:
> > >
> > > > I think this may have come up in another PR already (have to look for
> > > it).
> > > > But maybe we could maintain our flexibility in quick-dev by
> installing
> > > the
> > > > sensors and not starting them until we need them. I think it's useful
> > to
> > > > have a quick "genuine" e2e testing environment that doesn't require
> > > running
> > > > through a full install. I'm also not opposed to extracting the
> > > integration
> > > > test functionality into general purpose data generators.
> > > >
> > > > On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen  >
> > wrote:
> > > >
> > > > > To Jon's point, I think it would be useful to have a Demo box that
> > uses
> > > > > generators to produce 3 or 4 types of telemetry that shows up in
> the
> > > > Metron
> > > > > Dashboard. This box would be different from Quick-Dev in that
> > > everything
> > > > > starts automatically, so that a user just has to launch it and the
> > > should
> > > > > start seeing data in the Metron Dashboard right away. In fact, we
> > could
> > > > > even pre-load the Elasticsearch indices so that the user has more
> of
> > a
> > > > > history to mine when using the Demo box.
> > > > >
> > > > > On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com <
> zeo...@gmail.com >
> > > > > wrote:
> > > > >
> > > > > > +1 Ryan and Otto's comments.
> > > > > >
> > > > > > I also strongly think we need to make a demo environment easier,
> > but
> > > > that
> > > > > > should be different than quick-dev.
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler <
> > ottobackwa...@gmail.com 
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > - create scripts/utilities to easily run a topology 

Re: [DISCUSS] Improving quick-dev

[Kyle Richardson]

Sorry, I'm a bit late to the party on this one :).

+1 on the four builds Nick described. Each would be useful and purpose
built.

I like the idea of using Docker, especially for local development and quick
testing. Has anyone explored this? I'm envisioning very specific containers
so you could spin up only the components you're actively working on.
Certainly something I would be willing to put some cycles into if there is
interest.

-Kyle

On Fri, Oct 14, 2016 at 2:15 PM, Ryan Merriman  wrote:

> +1 I like it.  Just to clarify, the scripts to run Storm topologies locally
> in an IDE should be available independent of the environment running.  No
> need for a separate build/image.
>
> On Fri, Oct 14, 2016 at 9:12 AM, Otto Fowler 
> wrote:
>
> > Going forward, the Demo env and data would have implications for testing
> as
> > well ( gold data sets ) etc.
> >
> > On October 14, 2016 at 09:52:07, Nick Allen (n...@nickallen.org) wrote:
> >
> > I think based on everyone's input so far, we're describing 4 different
> > builds/images/tools that would each be intended to run on a standard
> > Mac/Linux/Windows laptop.
> >
> > Full Dev - A development environment that performs a full end-to-end
> > deployment of Metron. This is intended for developers working with
> > sensors, deployments, or validating how all Metron components interact
> with
> > one another.
> >
> >
> > - Starts from base Linux image
> > - Installs Hadoop-y components
> > - Installs Metron
> > - Installs Sensors
> > - Nothing started by default
> >
> > Quick Dev - An environment intended for the developer focusing on the
> > streaming components of Metron; parsing, enrichment, and indexing.
> >
> >
> > - Starts from base image of Linux + Hadoop-y components
> > - Installs Metron
> > - Installs "data generator" spouts
> > - Does not install sensors
> > - Nothing started by default
> >
> > Demo - An environment intended to introduce new users to Metron. The
> > environment should go from nothing to plenty of data in the Metron
> > dashboard in as little "boot" time as possible.
> >
> >
> > - Starts from a base image including Linux + Hadoop-y + Metron + Data
> > Generator Spouts pre-installed
> > - Pre-load Elasticsearch indices so the user has plenty of data to
> > investigate in the dashboard
> > - Does not install sensors
> > - Everything started by default
> >
> > Storm Local Cluster - Otto suggested some scripts/tooling to make it easy
> > to launch the core topologies on a local Storm cluster running on the
> host
> > OS.
> >
> >
> > I'd be interested to hear if this works for everyone and how this might
> > play into the Ambari mpack + RPM based deployment scheme.
> >
> >
> > On Fri, Oct 14, 2016 at 1:45 AM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > I think this may have come up in another PR already (have to look for
> > it).
> > > But maybe we could maintain our flexibility in quick-dev by installing
> > the
> > > sensors and not starting them until we need them. I think it's useful
> to
> > > have a quick "genuine" e2e testing environment that doesn't require
> > running
> > > through a full install. I'm also not opposed to extracting the
> > integration
> > > test functionality into general purpose data generators.
> > >
> > > On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen 
> wrote:
> > >
> > > > To Jon's point, I think it would be useful to have a Demo box that
> uses
> > > > generators to produce 3 or 4 types of telemetry that shows up in the
> > > Metron
> > > > Dashboard. This box would be different from Quick-Dev in that
> > everything
> > > > starts automatically, so that a user just has to launch it and the
> > should
> > > > start seeing data in the Metron Dashboard right away. In fact, we
> could
> > > > even pre-load the Elasticsearch indices so that the user has more of
> a
> > > > history to mine when using the Demo box.
> > > >
> > > > On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com 
> > > > wrote:
> > > >
> > > > > +1 Ryan and Otto's comments.
> > > > >
> > > > > I also strongly think we need to make a demo environment easier,
> but
> > > that
> > > > > should be different than quick-dev.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler <
> ottobackwa...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > - create scripts/utilities to easily run a topology locally in an
> > IDE
> > > > > > instead of in the VM
> > > > > >
> > > > > >
> > > > > >  THIS.
> > > > > >
> > > > > >
> > > > > > On October 13, 2016 at 12:36:45, Ryan Merriman (
> > merrim...@gmail.com)
> >
> > > > > > wrote:
> > > > > >
> > > > > > Working with the quick-dev vagrant VM recently left a lot to be
> > > > desired.
> > > > > > All forthcoming comments are made under the assumption that this
> VM
> > > is
> > > > > > intended for development purposes. If that is not true, I think
> we
> > > > 

Re: [DISCUSS] Improving quick-dev

[Ryan Merriman]

+1 I like it.  Just to clarify, the scripts to run Storm topologies locally
in an IDE should be available independent of the environment running.  No
need for a separate build/image.

On Fri, Oct 14, 2016 at 9:12 AM, Otto Fowler 
wrote:

> Going forward, the Demo env and data would have implications for testing as
> well ( gold data sets ) etc.
>
> On October 14, 2016 at 09:52:07, Nick Allen (n...@nickallen.org) wrote:
>
> I think based on everyone's input so far, we're describing 4 different
> builds/images/tools that would each be intended to run on a standard
> Mac/Linux/Windows laptop.
>
> Full Dev - A development environment that performs a full end-to-end
> deployment of Metron. This is intended for developers working with
> sensors, deployments, or validating how all Metron components interact with
> one another.
>
>
> - Starts from base Linux image
> - Installs Hadoop-y components
> - Installs Metron
> - Installs Sensors
> - Nothing started by default
>
> Quick Dev - An environment intended for the developer focusing on the
> streaming components of Metron; parsing, enrichment, and indexing.
>
>
> - Starts from base image of Linux + Hadoop-y components
> - Installs Metron
> - Installs "data generator" spouts
> - Does not install sensors
> - Nothing started by default
>
> Demo - An environment intended to introduce new users to Metron. The
> environment should go from nothing to plenty of data in the Metron
> dashboard in as little "boot" time as possible.
>
>
> - Starts from a base image including Linux + Hadoop-y + Metron + Data
> Generator Spouts pre-installed
> - Pre-load Elasticsearch indices so the user has plenty of data to
> investigate in the dashboard
> - Does not install sensors
> - Everything started by default
>
> Storm Local Cluster - Otto suggested some scripts/tooling to make it easy
> to launch the core topologies on a local Storm cluster running on the host
> OS.
>
>
> I'd be interested to hear if this works for everyone and how this might
> play into the Ambari mpack + RPM based deployment scheme.
>
>
> On Fri, Oct 14, 2016 at 1:45 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I think this may have come up in another PR already (have to look for
> it).
> > But maybe we could maintain our flexibility in quick-dev by installing
> the
> > sensors and not starting them until we need them. I think it's useful to
> > have a quick "genuine" e2e testing environment that doesn't require
> running
> > through a full install. I'm also not opposed to extracting the
> integration
> > test functionality into general purpose data generators.
> >
> > On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen  wrote:
> >
> > > To Jon's point, I think it would be useful to have a Demo box that uses
> > > generators to produce 3 or 4 types of telemetry that shows up in the
> > Metron
> > > Dashboard. This box would be different from Quick-Dev in that
> everything
> > > starts automatically, so that a user just has to launch it and the
> should
> > > start seeing data in the Metron Dashboard right away. In fact, we could
> > > even pre-load the Elasticsearch indices so that the user has more of a
> > > history to mine when using the Demo box.
> > >
> > > On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com 
> > > wrote:
> > >
> > > > +1 Ryan and Otto's comments.
> > > >
> > > > I also strongly think we need to make a demo environment easier, but
> > that
> > > > should be different than quick-dev.
> > > >
> > > > Jon
> > > >
> > > > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler  >
> > > > wrote:
> > > >
> > > > > - create scripts/utilities to easily run a topology locally in an
> IDE
> > > > > instead of in the VM
> > > > >
> > > > >
> > > > >  THIS.
> > > > >
> > > > >
> > > > > On October 13, 2016 at 12:36:45, Ryan Merriman (
> merrim...@gmail.com)
>
> > > > > wrote:
> > > > >
> > > > > Working with the quick-dev vagrant VM recently left a lot to be
> > > desired.
> > > > > All forthcoming comments are made under the assumption that this VM
> > is
> > > > > intended for development purposes. If that is not true, I think we
> > > should
> > > > > consider adding a VM for this purpose (or Docker containers?). Here
> > are
> > > > > the issues I ran into that I think can be improved:
> > > > >
> > > > > - had to upgrade VirtualBox from 5.0.16 to 5.0.20
> > > > > - had to update to the latest metron/hdp-base Vagrant box
> > > > > - takes forever to spin up
> > > > > - VM is constrained for resources making it unstable
> > > > > - spent a large amount of time troubleshooting sensors (no raw
> > messages
> > > > > in Kafka)
> > > > > - no easy way to debug topologies
> > > > >
> > > > > Fortunately I think we can make this a much better experience
> > without a
> > > > > major effort. Here are my ideas to do this:
> > > > >
> > > > > - update the prereqs for VirtualBox
> > > > > - add a check for the 

Re: [DISCUSS] Improving quick-dev

[Otto Fowler]

Going forward, the Demo env and data would have implications for testing as
well ( gold data sets ) etc.

On October 14, 2016 at 09:52:07, Nick Allen (n...@nickallen.org) wrote:

I think based on everyone's input so far, we're describing 4 different
builds/images/tools that would each be intended to run on a standard
Mac/Linux/Windows laptop.

Full Dev - A development environment that performs a full end-to-end
deployment of Metron. This is intended for developers working with
sensors, deployments, or validating how all Metron components interact with
one another.


- Starts from base Linux image
- Installs Hadoop-y components
- Installs Metron
- Installs Sensors
- Nothing started by default

Quick Dev - An environment intended for the developer focusing on the
streaming components of Metron; parsing, enrichment, and indexing.


- Starts from base image of Linux + Hadoop-y components
- Installs Metron
- Installs "data generator" spouts
- Does not install sensors
- Nothing started by default

Demo - An environment intended to introduce new users to Metron. The
environment should go from nothing to plenty of data in the Metron
dashboard in as little "boot" time as possible.


- Starts from a base image including Linux + Hadoop-y + Metron + Data
Generator Spouts pre-installed
- Pre-load Elasticsearch indices so the user has plenty of data to
investigate in the dashboard
- Does not install sensors
- Everything started by default

Storm Local Cluster - Otto suggested some scripts/tooling to make it easy
to launch the core topologies on a local Storm cluster running on the host
OS.


I'd be interested to hear if this works for everyone and how this might
play into the Ambari mpack + RPM based deployment scheme.


On Fri, Oct 14, 2016 at 1:45 AM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I think this may have come up in another PR already (have to look for
it).
> But maybe we could maintain our flexibility in quick-dev by installing
the
> sensors and not starting them until we need them. I think it's useful to
> have a quick "genuine" e2e testing environment that doesn't require
running
> through a full install. I'm also not opposed to extracting the
integration
> test functionality into general purpose data generators.
>
> On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen  wrote:
>
> > To Jon's point, I think it would be useful to have a Demo box that uses
> > generators to produce 3 or 4 types of telemetry that shows up in the
> Metron
> > Dashboard. This box would be different from Quick-Dev in that
everything
> > starts automatically, so that a user just has to launch it and the
should
> > start seeing data in the Metron Dashboard right away. In fact, we could
> > even pre-load the Elasticsearch indices so that the user has more of a
> > history to mine when using the Demo box.
> >
> > On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com 
> > wrote:
> >
> > > +1 Ryan and Otto's comments.
> > >
> > > I also strongly think we need to make a demo environment easier, but
> that
> > > should be different than quick-dev.
> > >
> > > Jon
> > >
> > > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler 
> > > wrote:
> > >
> > > > - create scripts/utilities to easily run a topology locally in an
IDE
> > > > instead of in the VM
> > > >
> > > >
> > > >  THIS.
> > > >
> > > >
> > > > On October 13, 2016 at 12:36:45, Ryan Merriman (merrim...@gmail.com)

> > > > wrote:
> > > >
> > > > Working with the quick-dev vagrant VM recently left a lot to be
> > desired.
> > > > All forthcoming comments are made under the assumption that this VM
> is
> > > > intended for development purposes. If that is not true, I think we
> > should
> > > > consider adding a VM for this purpose (or Docker containers?). Here
> are
> > > > the issues I ran into that I think can be improved:
> > > >
> > > > - had to upgrade VirtualBox from 5.0.16 to 5.0.20
> > > > - had to update to the latest metron/hdp-base Vagrant box
> > > > - takes forever to spin up
> > > > - VM is constrained for resources making it unstable
> > > > - spent a large amount of time troubleshooting sensors (no raw
> messages
> > > > in Kafka)
> > > > - no easy way to debug topologies
> > > >
> > > > Fortunately I think we can make this a much better experience
> without a
> > > > major effort. Here are my ideas to do this:
> > > >
> > > > - update the prereqs for VirtualBox
> > > > - add a check for the appropriate base box version (Jira has
already
> > > > been created https://issues.apache.org/jira/browse/METRON-497)
> > > > - don't install any sensors and replace them with a data generator
> that
> > > > just loops through sample data and emits to Kafka (could also be
used
> > to
> > > > replay and troubleshoot edge cases)
> > > > - everything in monit is off by default except for ES or other
> critical
> > > > services
> > > > - create scripts/utilities to easily run a topology locally in an
IDE
> > > > 

Re: [DISCUSS] Improving quick-dev

[Nick Allen]

I think based on everyone's input so far, we're describing 4 different
builds/images/tools that would each be intended to run on a standard
Mac/Linux/Windows laptop.

Full Dev - A development environment that performs a full end-to-end
deployment of Metron.  This is intended for developers working with
sensors, deployments, or validating how all Metron components interact with
one another.


   - Starts from base Linux image
  - Installs Hadoop-y components
  - Installs Metron
  - Installs Sensors
  - Nothing started by default

Quick Dev - An environment intended for the developer focusing on the
streaming components of Metron; parsing, enrichment, and indexing.


   - Starts from base image of Linux + Hadoop-y components
  - Installs Metron
  - Installs "data generator" spouts
  - Does not install sensors
  - Nothing started by default

Demo - An environment intended to introduce new users to Metron.  The
environment should go from nothing to plenty of data in the Metron
dashboard in as little "boot" time as possible.


   - Starts from a base image including Linux + Hadoop-y + Metron + Data
  Generator Spouts pre-installed
  - Pre-load Elasticsearch indices so the user has plenty of data to
  investigate in the dashboard
  - Does not install sensors
  - Everything started by default

Storm Local Cluster - Otto suggested some scripts/tooling to make it easy
to launch the core topologies on a local Storm cluster running on the host
OS.


 I'd be interested to hear if this works for everyone and how this might
play into the Ambari mpack + RPM based deployment scheme.


On Fri, Oct 14, 2016 at 1:45 AM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I think this may have come up in another PR already (have to look for it).
> But maybe we could maintain our flexibility in quick-dev by installing the
> sensors and not starting them until we need them. I think it's useful to
> have a quick "genuine" e2e testing environment that doesn't require running
> through a full install. I'm also not opposed to extracting the integration
> test functionality into general purpose data generators.
>
> On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen  wrote:
>
> > To Jon's point, I think it would be useful to have a Demo box that uses
> > generators to produce 3 or 4 types of telemetry that shows up in the
> Metron
> > Dashboard.  This box would be different from Quick-Dev in that everything
> > starts automatically, so that a user just has to launch it and the should
> > start seeing data in the Metron Dashboard right away.  In fact, we could
> > even pre-load the Elasticsearch indices so that the user has more of a
> > history to mine when using the Demo box.
> >
> > On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com 
> > wrote:
> >
> > > +1 Ryan and Otto's comments.
> > >
> > > I also strongly think we need to make a demo environment easier, but
> that
> > > should be different than quick-dev.
> > >
> > > Jon
> > >
> > > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler 
> > > wrote:
> > >
> > > > - create scripts/utilities to easily run a topology locally in an IDE
> > > > instead of in the VM
> > > >
> > > >
> > > >  THIS.
> > > >
> > > >
> > > > On October 13, 2016 at 12:36:45, Ryan Merriman (merrim...@gmail.com)
> > > > wrote:
> > > >
> > > > Working with the quick-dev vagrant VM recently left a lot to be
> > desired.
> > > > All forthcoming comments are made under the assumption that this VM
> is
> > > > intended for development purposes. If that is not true, I think we
> > should
> > > > consider adding a VM for this purpose (or Docker containers?). Here
> are
> > > > the issues I ran into that I think can be improved:
> > > >
> > > > - had to upgrade VirtualBox from 5.0.16 to 5.0.20
> > > > - had to update to the latest metron/hdp-base Vagrant box
> > > > - takes forever to spin up
> > > > - VM is constrained for resources making it unstable
> > > > - spent a large amount of time troubleshooting sensors (no raw
> messages
> > > > in Kafka)
> > > > - no easy way to debug topologies
> > > >
> > > > Fortunately I think we can make this a much better experience
> without a
> > > > major effort. Here are my ideas to do this:
> > > >
> > > > - update the prereqs for VirtualBox
> > > > - add a check for the appropriate base box version (Jira has already
> > > > been created https://issues.apache.org/jira/browse/METRON-497)
> > > > - don't install any sensors and replace them with a data generator
> that
> > > > just loops through sample data and emits to Kafka (could also be used
> > to
> > > > replay and troubleshoot edge cases)
> > > > - everything in monit is off by default except for ES or other
> critical
> > > > services
> > > > - create scripts/utilities to easily run a topology locally in an IDE
> > > > instead of in the VM
> > > > - improved documentation with examples of how to run and 

Re: [DISCUSS] Improving quick-dev

[Michael Miklavcic]

I think this may have come up in another PR already (have to look for it).
But maybe we could maintain our flexibility in quick-dev by installing the
sensors and not starting them until we need them. I think it's useful to
have a quick "genuine" e2e testing environment that doesn't require running
through a full install. I'm also not opposed to extracting the integration
test functionality into general purpose data generators.

On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen  wrote:

> To Jon's point, I think it would be useful to have a Demo box that uses
> generators to produce 3 or 4 types of telemetry that shows up in the Metron
> Dashboard.  This box would be different from Quick-Dev in that everything
> starts automatically, so that a user just has to launch it and the should
> start seeing data in the Metron Dashboard right away.  In fact, we could
> even pre-load the Elasticsearch indices so that the user has more of a
> history to mine when using the Demo box.
>
> On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com 
> wrote:
>
> > +1 Ryan and Otto's comments.
> >
> > I also strongly think we need to make a demo environment easier, but that
> > should be different than quick-dev.
> >
> > Jon
> >
> > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler 
> > wrote:
> >
> > > - create scripts/utilities to easily run a topology locally in an IDE
> > > instead of in the VM
> > >
> > >
> > >  THIS.
> > >
> > >
> > > On October 13, 2016 at 12:36:45, Ryan Merriman (merrim...@gmail.com)
> > > wrote:
> > >
> > > Working with the quick-dev vagrant VM recently left a lot to be
> desired.
> > > All forthcoming comments are made under the assumption that this VM is
> > > intended for development purposes. If that is not true, I think we
> should
> > > consider adding a VM for this purpose (or Docker containers?). Here are
> > > the issues I ran into that I think can be improved:
> > >
> > > - had to upgrade VirtualBox from 5.0.16 to 5.0.20
> > > - had to update to the latest metron/hdp-base Vagrant box
> > > - takes forever to spin up
> > > - VM is constrained for resources making it unstable
> > > - spent a large amount of time troubleshooting sensors (no raw messages
> > > in Kafka)
> > > - no easy way to debug topologies
> > >
> > > Fortunately I think we can make this a much better experience without a
> > > major effort. Here are my ideas to do this:
> > >
> > > - update the prereqs for VirtualBox
> > > - add a check for the appropriate base box version (Jira has already
> > > been created https://issues.apache.org/jira/browse/METRON-497)
> > > - don't install any sensors and replace them with a data generator that
> > > just loops through sample data and emits to Kafka (could also be used
> to
> > > replay and troubleshoot edge cases)
> > > - everything in monit is off by default except for ES or other critical
> > > services
> > > - create scripts/utilities to easily run a topology locally in an IDE
> > > instead of in the VM
> > > - improved documentation with examples of how to run and troubleshoot
> > > topologies
> > >
> > > Is this a worthwhile effort? I think this would also give users an
> easier
> > > path to demonstrate or tour Metron's capabilities. Are there any other
> > > improvements people would like to see? Should we wait for Docker?
> > > Thoughts?
> > >
> > > Ryan Merriman
> > >
> > --
> >
> > Jon
> >
>
>
>
> --
> Nick Allen 
>

Re: [DISCUSS] Improving quick-dev

[Nick Allen]

To Jon's point, I think it would be useful to have a Demo box that uses
generators to produce 3 or 4 types of telemetry that shows up in the Metron
Dashboard.  This box would be different from Quick-Dev in that everything
starts automatically, so that a user just has to launch it and the should
start seeing data in the Metron Dashboard right away.  In fact, we could
even pre-load the Elasticsearch indices so that the user has more of a
history to mine when using the Demo box.

On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com  wrote:

> +1 Ryan and Otto's comments.
>
> I also strongly think we need to make a demo environment easier, but that
> should be different than quick-dev.
>
> Jon
>
> On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler 
> wrote:
>
> > - create scripts/utilities to easily run a topology locally in an IDE
> > instead of in the VM
> >
> >
> >  THIS.
> >
> >
> > On October 13, 2016 at 12:36:45, Ryan Merriman (merrim...@gmail.com)
> > wrote:
> >
> > Working with the quick-dev vagrant VM recently left a lot to be desired.
> > All forthcoming comments are made under the assumption that this VM is
> > intended for development purposes. If that is not true, I think we should
> > consider adding a VM for this purpose (or Docker containers?). Here are
> > the issues I ran into that I think can be improved:
> >
> > - had to upgrade VirtualBox from 5.0.16 to 5.0.20
> > - had to update to the latest metron/hdp-base Vagrant box
> > - takes forever to spin up
> > - VM is constrained for resources making it unstable
> > - spent a large amount of time troubleshooting sensors (no raw messages
> > in Kafka)
> > - no easy way to debug topologies
> >
> > Fortunately I think we can make this a much better experience without a
> > major effort. Here are my ideas to do this:
> >
> > - update the prereqs for VirtualBox
> > - add a check for the appropriate base box version (Jira has already
> > been created https://issues.apache.org/jira/browse/METRON-497)
> > - don't install any sensors and replace them with a data generator that
> > just loops through sample data and emits to Kafka (could also be used to
> > replay and troubleshoot edge cases)
> > - everything in monit is off by default except for ES or other critical
> > services
> > - create scripts/utilities to easily run a topology locally in an IDE
> > instead of in the VM
> > - improved documentation with examples of how to run and troubleshoot
> > topologies
> >
> > Is this a worthwhile effort? I think this would also give users an easier
> > path to demonstrate or tour Metron's capabilities. Are there any other
> > improvements people would like to see? Should we wait for Docker?
> > Thoughts?
> >
> > Ryan Merriman
> >
> --
>
> Jon
>



-- 
Nick Allen 

Re: [DISCUSS] Improving quick-dev

[Nick Allen]

I really like the idea to replace real sensors in Quick-Dev with data
generators; aka spouts that spit-out canned data.  The pcap replay
mechanism is fairly resource intensive, not to mention all of the sensors
like Bro, Snort, etc.  Removing these should give us significantly more
head room.

At the same time, it is going to be useful in some cases to have a single
node that can run end-to-end with sensors.  For example, I still want to
have a box that I can drop in different pcap files and capture telemetry
for testing.  I think we should continue to maintain Full-Dev with the
current sensor suite.

In addition, having these data generators can help with performance
testing, if we add the right toggles and switches.  Like the ability to
control the throughput, etc.


On Thu, Oct 13, 2016 at 12:36 PM, Ryan Merriman  wrote:

> Working with the quick-dev vagrant VM recently left a lot to be desired.
> All forthcoming comments are made under the assumption that this VM is
> intended for development purposes.  If that is not true, I think we should
> consider adding a VM for this purpose (or Docker containers?).  Here are
> the issues I ran into that I think can be improved:
>
>- had to upgrade VirtualBox from 5.0.16 to 5.0.20
>- had to update to the latest metron/hdp-base Vagrant box
>- takes forever to spin up
>- VM is constrained for resources making it unstable
>- spent a large amount of time troubleshooting sensors (no raw messages
>in Kafka)
>- no easy way to debug topologies
>
> Fortunately I think we can make this a much better experience without a
> major effort.  Here are my ideas to do this:
>
>- update the prereqs for VirtualBox
>- add a check for the appropriate base box version (Jira has already
>been created https://issues.apache.org/jira/browse/METRON-497)
>- don't install any sensors and replace them with a data generator that
>just loops through sample data and emits to Kafka (could also be used to
>replay and troubleshoot edge cases)
>- everything in monit is off by default except for ES or other critical
>services
>- create scripts/utilities to easily run a topology locally in an IDE
>instead of in the VM
>- improved documentation with examples of how to run and troubleshoot
>topologies
>
> Is this a worthwhile effort?  I think this would also give users an easier
> path to demonstrate or tour Metron's capabilities.  Are there any other
> improvements people would like to see?  Should we wait for Docker?
> Thoughts?
>
> Ryan Merriman
>



-- 
Nick Allen 

Re: [DISCUSS] Improving quick-dev

[Casey Stella]

You could.  This was a consideration when creating the IntegrationTest
infrastructure.There are some challenges here, though, with hbase and
storm coexisting in the same jvm.  I do not know if this is still an issue
with storm 1.0+.

On Thu, Oct 13, 2016 at 2:26 PM, Otto Fowler 
wrote:

> Could we use the IntegrationTest components to do this?
>
>
>
> On October 13, 2016 at 14:04:54, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
>
> +1 Ryan and Otto's comments.
>
> I also strongly think we need to make a demo environment easier, but that
> should be different than quick-dev.
>
> Jon
>
> On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler 
> wrote:
>
> > - create scripts/utilities to easily run a topology locally in an IDE
> > instead of in the VM
> >
> >
> >  THIS.
> >
> >
> > On October 13, 2016 at 12:36:45, Ryan Merriman (merrim...@gmail.com)
> > wrote:
> >
> > Working with the quick-dev vagrant VM recently left a lot to be desired.
> > All forthcoming comments are made under the assumption that this VM is
> > intended for development purposes. If that is not true, I think we should
> > consider adding a VM for this purpose (or Docker containers?). Here are
> > the issues I ran into that I think can be improved:
> >
> > - had to upgrade VirtualBox from 5.0.16 to 5.0.20
> > - had to update to the latest metron/hdp-base Vagrant box
> > - takes forever to spin up
> > - VM is constrained for resources making it unstable
> > - spent a large amount of time troubleshooting sensors (no raw messages
> > in Kafka)
> > - no easy way to debug topologies
> >
> > Fortunately I think we can make this a much better experience without a
> > major effort. Here are my ideas to do this:
> >
> > - update the prereqs for VirtualBox
> > - add a check for the appropriate base box version (Jira has already
> > been created https://issues.apache.org/jira/browse/METRON-497)
> > - don't install any sensors and replace them with a data generator that
> > just loops through sample data and emits to Kafka (could also be used to
> > replay and troubleshoot edge cases)
> > - everything in monit is off by default except for ES or other critical
> > services
> > - create scripts/utilities to easily run a topology locally in an IDE
> > instead of in the VM
> > - improved documentation with examples of how to run and troubleshoot
> > topologies
> >
> > Is this a worthwhile effort? I think this would also give users an easier
> > path to demonstrate or tour Metron's capabilities. Are there any other
> > improvements people would like to see? Should we wait for Docker?
> > Thoughts?
> >
> > Ryan Merriman
> >
> --
>
> Jon
>

Re: [DISCUSS] Improving quick-dev

[zeo...@gmail.com]

+1 Ryan and Otto's comments.

I also strongly think we need to make a demo environment easier, but that
should be different than quick-dev.

Jon

On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler  wrote:

> - create scripts/utilities to easily run a topology locally in an IDE
> instead of in the VM
>
>
>  THIS.
>
>
> On October 13, 2016 at 12:36:45, Ryan Merriman (merrim...@gmail.com)
> wrote:
>
> Working with the quick-dev vagrant VM recently left a lot to be desired.
> All forthcoming comments are made under the assumption that this VM is
> intended for development purposes. If that is not true, I think we should
> consider adding a VM for this purpose (or Docker containers?). Here are
> the issues I ran into that I think can be improved:
>
> - had to upgrade VirtualBox from 5.0.16 to 5.0.20
> - had to update to the latest metron/hdp-base Vagrant box
> - takes forever to spin up
> - VM is constrained for resources making it unstable
> - spent a large amount of time troubleshooting sensors (no raw messages
> in Kafka)
> - no easy way to debug topologies
>
> Fortunately I think we can make this a much better experience without a
> major effort. Here are my ideas to do this:
>
> - update the prereqs for VirtualBox
> - add a check for the appropriate base box version (Jira has already
> been created https://issues.apache.org/jira/browse/METRON-497)
> - don't install any sensors and replace them with a data generator that
> just loops through sample data and emits to Kafka (could also be used to
> replay and troubleshoot edge cases)
> - everything in monit is off by default except for ES or other critical
> services
> - create scripts/utilities to easily run a topology locally in an IDE
> instead of in the VM
> - improved documentation with examples of how to run and troubleshoot
> topologies
>
> Is this a worthwhile effort? I think this would also give users an easier
> path to demonstrate or tour Metron's capabilities. Are there any other
> improvements people would like to see? Should we wait for Docker?
> Thoughts?
>
> Ryan Merriman
>
-- 

Jon

Re: [DISCUSS] Improving quick-dev

[Otto Fowler]

- create scripts/utilities to easily run a topology locally in an IDE
instead of in the VM


 THIS.


On October 13, 2016 at 12:36:45, Ryan Merriman (merrim...@gmail.com) wrote:

Working with the quick-dev vagrant VM recently left a lot to be desired.
All forthcoming comments are made under the assumption that this VM is
intended for development purposes. If that is not true, I think we should
consider adding a VM for this purpose (or Docker containers?). Here are
the issues I ran into that I think can be improved:

- had to upgrade VirtualBox from 5.0.16 to 5.0.20
- had to update to the latest metron/hdp-base Vagrant box
- takes forever to spin up
- VM is constrained for resources making it unstable
- spent a large amount of time troubleshooting sensors (no raw messages
in Kafka)
- no easy way to debug topologies

Fortunately I think we can make this a much better experience without a
major effort. Here are my ideas to do this:

- update the prereqs for VirtualBox
- add a check for the appropriate base box version (Jira has already
been created https://issues.apache.org/jira/browse/METRON-497)
- don't install any sensors and replace them with a data generator that
just loops through sample data and emits to Kafka (could also be used to
replay and troubleshoot edge cases)
- everything in monit is off by default except for ES or other critical
services
- create scripts/utilities to easily run a topology locally in an IDE
instead of in the VM
- improved documentation with examples of how to run and troubleshoot
topologies

Is this a worthwhile effort? I think this would also give users an easier
path to demonstrate or tour Metron's capabilities. Are there any other
improvements people would like to see? Should we wait for Docker?
Thoughts?

Ryan Merriman