Re: [DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread Otto Fowler
https://github.com/apache/metron-bro-plugin-kafka/tree/master/docker On March 8, 2019 at 14:28:20, zeo...@gmail.com (zeo...@gmail.com) wrote: So most importantly I want to make sure to give Otto credit for being the one who cleaned up the rudimentary testing steps we had for testing the plugin

Re: [DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread zeo...@gmail.com
So most importantly I want to make sure to give Otto credit for being the one who cleaned up the rudimentary testing steps we had for testing the plugin and turned it into the docker end to end. Right now we manually run the tests, as there were a few follow-ons we needed to work through before

Re: [DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread Otto Fowler
I believe that the TestContainers allows the ide case On March 8, 2019 at 11:38:24, Michael Miklavcic (michael.miklav...@gmail.com) wrote: I'm -1 on #1 unless there's some desperately compelling reason to go that route. It would be a regression in our test coverage, and at that point it's

Re: [DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread Simon Elliston Ball
The Docker option sounds like a much better and cleaner option for integration testing (closer to real too). My one question would be whether this would significantly increase test run time, and whether that would need Travis changes? Either way, the docker option sounds best. Simon > On 8

Re: [DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread Michael Miklavcic
I'm -1 on #1 unless there's some desperately compelling reason to go that route. It would be a regression in our test coverage, and at that point it's really just duplicating our unit tests as opposed to checking our integration. I'm good with 3. Gating factors for a successful implementation

Re: [DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread Michael Miklavcic
Jon, I didn't get a chance to review your PR, but it immediately came to mind wrt this task - did you guys have that working in Travis with Maven? What's the container lifecycle you came up with look like? On Fri, Mar 8, 2019 at 8:30 AM zeo...@gmail.com wrote: > +1 to option 3 on both. Also

Re: [DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread Otto Fowler
I think I have mentioned it before, but https://www.testcontainers.org could be a viable approach for this methodology (3). I would think it would be worth looking at. On March 8, 2019 at 09:47:54, Ryan Merriman (merrim...@gmail.com) wrote: I have been researching the effort involved to

Re: [DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread zeo...@gmail.com
+1 to option 3 on both. Also strongly in favor of Docker. We recently took a similar approach in metron-bro-plugin-kafka as well (link ) to do end to end testing. Jon On Fri, Mar 8, 2019 at 9:53 AM Nick Allen wrote: > +1

Re: [DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread Nick Allen
+1 for option 3. I am in favor of using Docker for the integration tests for all the reasons that you mentioned. On Fri, Mar 8, 2019 at 9:47 AM Ryan Merriman wrote: > I have been researching the effort involved to upgrade to HDP 3. Along the > way I've found a couple challenging issues that

[DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread Ryan Merriman
I have been researching the effort involved to upgrade to HDP 3. Along the way I've found a couple challenging issues that we will need to solve, both involving our integration testing strategy. The first issue is Kafka. We are moving from 0.10.0 to 2.0.0 and there have been significant changes