I really like this On Oct 1, 2016 5:03 PM, "Ryan Merriman" <[email protected]> wrote:
> I don't think this will be an issue. The remote debugging strategy should > work if you want to go that route but I would just run the topology in > local mode (that's how the integration tests work now). The Storm topology > continues to runs locally in your IDE while all the other components that > were in-memory components (HBase, Kafka, etc) now run in Docker > containers. We can continue to leverage > https://github.com/apache/incubator-metron/blob/master/ > metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/ > integration/components/ParserTopologyComponent.java > for parser topologies and the --local option for flux-based topologies. > > On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen <[email protected]> wrote: > > > It's definitely possible to debug in a Docker environment using a remote > > debugger. I remember reading something along these lines a few weeks > ago. > > > > https://blog.docker.com/2016/09/java-development-using-docker/ > > > > On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella <[email protected]> wrote: > > > > > I agree with that concern. Being able to debug a running topology has > > > helped in so many circumstances. Not sure how to accomplish this in a > > > dockerized environment. > > > > > > On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic < > > > [email protected]> > > > wrote: > > > > > > > Any ideas on how debugging works when leveraging Docker? In spite of > > the > > > > classpath troubles, one of the benefits of the current single-JVM > > > approach > > > > is that you can easily debug, set break points, etc within an IDE. Is > > > that > > > > still doable? Seems like there might need to be some remote debugging > > > magic > > > > done to accomplish this. > > > > > > > > On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <[email protected]> > > > wrote: > > > > > > > > > I agree with making an effort to create containers. I suggest > doing > > it > > > > on > > > > > a feature branch until we are feature complete and are able to > > migrate > > > > our > > > > > integration tests into a dockerized environment. > > > > > > > > > > 30.09.2016, 14:52, "Nick Allen" <[email protected]>: > > > > > > Relieve dependency version conflict issues? Umm, yes please. > I'll > > > > take a > > > > > > second helping of that too. > > > > > > > > > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > >> I would like to open up a discussion around creating Docker > > images > > > > for > > > > > >> Metron. Having this available would provide a leaner > alternative > > > to > > > > > the > > > > > >> ansible/vagrant environment for development tesing (and even > > > demoing > > > > > or > > > > > >> exploring features). It could also relieve some of the > > dependency > > > > > version > > > > > >> conflict issues that we've been experiencing when running > > > > integration > > > > > tests > > > > > >> in a single JVM. > > > > > >> > > > > > >> I would suggest the initial version be intended only for > > > development > > > > > and > > > > > >> testing purposes. The general approach could be to create an > > image > > > > for > > > > > >> each service we depend on and use something like Docker > compose > > to > > > > > package > > > > > >> them together. A Dockerfile would either install the service > > from > > > > > scratch > > > > > >> or extend a community image then add any Metron related > > > dependencies > > > > > or > > > > > >> configurations on top. The metron-deployment project code > could > > be > > > > > used as > > > > > >> a guide. > > > > > >> > > > > > >> I would like to see these images added initially to support > > > > > development and > > > > > >> testing: > > > > > >> > > > > > >> - Kafka with topics preconfigured > > > > > >> - Storm with metron topology assets installed > > > > > >> - Zookeeper with paths created and sample configs loaded > > > > > >> - HBase with sample enrichments and threat intel loaded > > > > > >> - Elasticsearch configured for Metron > > > > > >> - MySQL with databases/tables/users created and geo data > > loaded > > > > > >> > > > > > >> Other images that could also be useful: > > > > > >> > > > > > >> - Images for each sensor > > > > > >> - Ambari? > > > > > >> - Solr > > > > > >> > > > > > >> Looking forward to hearing what everyone thinks. > > > > > >> > > > > > >> Ryan Merriman > > > > > > > > > > > > -- > > > > > > Nick Allen <[email protected]> > > > > > > > > > > > > > > > ------------------- > > > > > Thank you, > > > > > > > > > > James Sirota > > > > > PPMC- Apache Metron (Incubating) > > > > > jsirota AT apache DOT org > > > > > > > > > > 30.09.2016, 14:52, "Nick Allen" <[email protected]>: > > > > > > Relieve dependency version conflict issues? Umm, yes please. I'll > > > take > > > > a > > > > > > second helping of that too. > > > > > > > > > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman < > > [email protected]> > > > > > wrote: > > > > > > > > > > > >> I would like to open up a discussion around creating Docker > > images > > > > for > > > > > >> Metron. Having this available would provide a leaner > alternative > > to > > > > the > > > > > >> ansible/vagrant environment for development tesing (and even > > > demoing > > > > or > > > > > >> exploring features). It could also relieve some of the > dependency > > > > > version > > > > > >> conflict issues that we've been experiencing when running > > > integration > > > > > tests > > > > > >> in a single JVM. > > > > > >> > > > > > >> I would suggest the initial version be intended only for > > > development > > > > > and > > > > > >> testing purposes. The general approach could be to create an > > image > > > > for > > > > > >> each service we depend on and use something like Docker compose > > to > > > > > package > > > > > >> them together. A Dockerfile would either install the service > from > > > > > scratch > > > > > >> or extend a community image then add any Metron related > > > dependencies > > > > or > > > > > >> configurations on top. The metron-deployment project code could > > be > > > > > used as > > > > > >> a guide. > > > > > >> > > > > > >> I would like to see these images added initially to support > > > > > development and > > > > > >> testing: > > > > > >> > > > > > >> - Kafka with topics preconfigured > > > > > >> - Storm with metron topology assets installed > > > > > >> - Zookeeper with paths created and sample configs loaded > > > > > >> - HBase with sample enrichments and threat intel loaded > > > > > >> - Elasticsearch configured for Metron > > > > > >> - MySQL with databases/tables/users created and geo data > > loaded > > > > > >> > > > > > >> Other images that could also be useful: > > > > > >> > > > > > >> - Images for each sensor > > > > > >> - Ambari? > > > > > >> - Solr > > > > > >> > > > > > >> Looking forward to hearing what everyone thinks. > > > > > >> > > > > > >> Ryan Merriman > > > > > > > > > > > > -- > > > > > > Nick Allen <[email protected]> > > > > > > > > > > ------------------- > > > > > Thank you, > > > > > > > > > > James Sirota > > > > > PPMC- Apache Metron (Incubating) > > > > > jsirota AT apache DOT org > > > > > > > > > > > > > > > > > > > > -- > > Nick Allen <[email protected]> > > >
