At least on the Storm side, remote debugging can be problematic. If you pause execution of a worker process, it will stop heart beating and supervisor will kill it. Just something to be aware of.
I think we might have a flag to disable that behavior, but I'd have to check. I usually use local mode for debugging. -Taylor > On Oct 1, 2016, at 5:03 PM, Ryan Merriman <[email protected]> wrote: > > I don't think this will be an issue. The remote debugging strategy should > work if you want to go that route but I would just run the topology in > local mode (that's how the integration tests work now). The Storm topology > continues to runs locally in your IDE while all the other components that > were in-memory components (HBase, Kafka, etc) now run in Docker > containers. We can continue to leverage > https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java > for parser topologies and the --local option for flux-based topologies. > >> On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen <[email protected]> wrote: >> >> It's definitely possible to debug in a Docker environment using a remote >> debugger. I remember reading something along these lines a few weeks ago. >> >> https://blog.docker.com/2016/09/java-development-using-docker/ >> >>> On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella <[email protected]> wrote: >>> >>> I agree with that concern. Being able to debug a running topology has >>> helped in so many circumstances. Not sure how to accomplish this in a >>> dockerized environment. >>> >>> On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic < >>> [email protected]> >>> wrote: >>> >>>> Any ideas on how debugging works when leveraging Docker? In spite of >> the >>>> classpath troubles, one of the benefits of the current single-JVM >>> approach >>>> is that you can easily debug, set break points, etc within an IDE. Is >>> that >>>> still doable? Seems like there might need to be some remote debugging >>> magic >>>> done to accomplish this. >>>> >>>>> On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <[email protected]> >>>> wrote: >>>> >>>>> I agree with making an effort to create containers. I suggest doing >> it >>>> on >>>>> a feature branch until we are feature complete and are able to >> migrate >>>> our >>>>> integration tests into a dockerized environment. >>>>> >>>>> 30.09.2016, 14:52, "Nick Allen" <[email protected]>: >>>>>> Relieve dependency version conflict issues? Umm, yes please. I'll >>>> take a >>>>>> second helping of that too. >>>>>> >>>>>> On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman < >> [email protected] >>>> >>>>> wrote: >>>>>> >>>>>>> I would like to open up a discussion around creating Docker >> images >>>> for >>>>>>> Metron. Having this available would provide a leaner alternative >>> to >>>>> the >>>>>>> ansible/vagrant environment for development tesing (and even >>> demoing >>>>> or >>>>>>> exploring features). It could also relieve some of the >> dependency >>>>> version >>>>>>> conflict issues that we've been experiencing when running >>>> integration >>>>> tests >>>>>>> in a single JVM. >>>>>>> >>>>>>> I would suggest the initial version be intended only for >>> development >>>>> and >>>>>>> testing purposes. The general approach could be to create an >> image >>>> for >>>>>>> each service we depend on and use something like Docker compose >> to >>>>> package >>>>>>> them together. A Dockerfile would either install the service >> from >>>>> scratch >>>>>>> or extend a community image then add any Metron related >>> dependencies >>>>> or >>>>>>> configurations on top. The metron-deployment project code could >> be >>>>> used as >>>>>>> a guide. >>>>>>> >>>>>>> I would like to see these images added initially to support >>>>> development and >>>>>>> testing: >>>>>>> >>>>>>> - Kafka with topics preconfigured >>>>>>> - Storm with metron topology assets installed >>>>>>> - Zookeeper with paths created and sample configs loaded >>>>>>> - HBase with sample enrichments and threat intel loaded >>>>>>> - Elasticsearch configured for Metron >>>>>>> - MySQL with databases/tables/users created and geo data >> loaded >>>>>>> >>>>>>> Other images that could also be useful: >>>>>>> >>>>>>> - Images for each sensor >>>>>>> - Ambari? >>>>>>> - Solr >>>>>>> >>>>>>> Looking forward to hearing what everyone thinks. >>>>>>> >>>>>>> Ryan Merriman >>>>>> >>>>>> -- >>>>>> Nick Allen <[email protected]> >>>>> >>>>> >>>>> ------------------- >>>>> Thank you, >>>>> >>>>> James Sirota >>>>> PPMC- Apache Metron (Incubating) >>>>> jsirota AT apache DOT org >>>>> >>>>> 30.09.2016, 14:52, "Nick Allen" <[email protected]>: >>>>>> Relieve dependency version conflict issues? Umm, yes please. I'll >>> take >>>> a >>>>>> second helping of that too. >>>>>> >>>>>> On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman < >> [email protected]> >>>>> wrote: >>>>>> >>>>>>> I would like to open up a discussion around creating Docker >> images >>>> for >>>>>>> Metron. Having this available would provide a leaner alternative >> to >>>> the >>>>>>> ansible/vagrant environment for development tesing (and even >>> demoing >>>> or >>>>>>> exploring features). It could also relieve some of the dependency >>>>> version >>>>>>> conflict issues that we've been experiencing when running >>> integration >>>>> tests >>>>>>> in a single JVM. >>>>>>> >>>>>>> I would suggest the initial version be intended only for >>> development >>>>> and >>>>>>> testing purposes. The general approach could be to create an >> image >>>> for >>>>>>> each service we depend on and use something like Docker compose >> to >>>>> package >>>>>>> them together. A Dockerfile would either install the service from >>>>> scratch >>>>>>> or extend a community image then add any Metron related >>> dependencies >>>> or >>>>>>> configurations on top. The metron-deployment project code could >> be >>>>> used as >>>>>>> a guide. >>>>>>> >>>>>>> I would like to see these images added initially to support >>>>> development and >>>>>>> testing: >>>>>>> >>>>>>> - Kafka with topics preconfigured >>>>>>> - Storm with metron topology assets installed >>>>>>> - Zookeeper with paths created and sample configs loaded >>>>>>> - HBase with sample enrichments and threat intel loaded >>>>>>> - Elasticsearch configured for Metron >>>>>>> - MySQL with databases/tables/users created and geo data >> loaded >>>>>>> >>>>>>> Other images that could also be useful: >>>>>>> >>>>>>> - Images for each sensor >>>>>>> - Ambari? >>>>>>> - Solr >>>>>>> >>>>>>> Looking forward to hearing what everyone thinks. >>>>>>> >>>>>>> Ryan Merriman >>>>>> >>>>>> -- >>>>>> Nick Allen <[email protected]> >>>>> >>>>> ------------------- >>>>> Thank you, >>>>> >>>>> James Sirota >>>>> PPMC- Apache Metron (Incubating) >>>>> jsirota AT apache DOT org >> >> >> >> -- >> Nick Allen <[email protected]> >>
