Re: [DISCUSS] Upgrading HBase and Kafka support

+1 to Docker and looking at testcontainers.org On Fri, Mar 8, 2019 at 3:03 PM Otto Fowler wrote: > https://github.com/apache/metron-bro-plugin-kafka/tree/master/docker > > > On March 8, 2019 at 14:28:20, zeo...@gmail.com (zeo...@gmail.com) wrote: > > So most importantly I want to make sure to gi

Re: [DISCUSS] Upgrading HBase and Kafka support

https://github.com/apache/metron-bro-plugin-kafka/tree/master/docker On March 8, 2019 at 14:28:20, zeo...@gmail.com (zeo...@gmail.com) wrote: So most importantly I want to make sure to give Otto credit for being the one who cleaned up the rudimentary testing steps we had for testing the plugin a

Re: [DISCUSS] Upgrading HBase and Kafka support

So most importantly I want to make sure to give Otto credit for being the one who cleaned up the rudimentary testing steps we had for testing the plugin and turned it into the docker end to end. Right now we manually run the tests, as there were a few follow-ons we needed to work through before it

Re: [DISCUSS] Upgrading HBase and Kafka support

I believe that the TestContainers allows the ide case On March 8, 2019 at 11:38:24, Michael Miklavcic (michael.miklav...@gmail.com) wrote: I'm -1 on #1 unless there's some desperately compelling reason to go that route. It would be a regression in our test coverage, and at that point it's really

Re: [DISCUSS] Upgrading HBase and Kafka support

The Docker option sounds like a much better and cleaner option for integration testing (closer to real too). My one question would be whether this would significantly increase test run time, and whether that would need Travis changes? Either way, the docker option sounds best. Simon > On 8 M

Re: [DISCUSS] Upgrading HBase and Kafka support

I'm -1 on #1 unless there's some desperately compelling reason to go that route. It would be a regression in our test coverage, and at that point it's really just duplicating our unit tests as opposed to checking our integration. I'm good with 3. Gating factors for a successful implementation woul

Re: [DISCUSS] Upgrading HBase and Kafka support

Jon, I didn't get a chance to review your PR, but it immediately came to mind wrt this task - did you guys have that working in Travis with Maven? What's the container lifecycle you came up with look like? On Fri, Mar 8, 2019 at 8:30 AM zeo...@gmail.com wrote: > +1 to option 3 on both. Also str

Re: [DISCUSS] Upgrading HBase and Kafka support

I think I have mentioned it before, but https://www.testcontainers.org could be a viable approach for this methodology (3). I would think it would be worth looking at. On March 8, 2019 at 09:47:54, Ryan Merriman (merrim...@gmail.com) wrote: I have been researching the effort involved to upgrade

Re: [DISCUSS] Upgrading HBase and Kafka support

+1 to option 3 on both. Also strongly in favor of Docker. We recently took a similar approach in metron-bro-plugin-kafka as well (link ) to do end to end testing. Jon On Fri, Mar 8, 2019 at 9:53 AM Nick Allen wrote: > +1 fo

Re: [DISCUSS] Upgrading HBase and Kafka support

+1 for option 3. I am in favor of using Docker for the integration tests for all the reasons that you mentioned. On Fri, Mar 8, 2019 at 9:47 AM Ryan Merriman wrote: > I have been researching the effort involved to upgrade to HDP 3. Along the > way I've found a couple challenging issues that we

[DISCUSS] Upgrading HBase and Kafka support

I have been researching the effort involved to upgrade to HDP 3. Along the way I've found a couple challenging issues that we will need to solve, both involving our integration testing strategy. The first issue is Kafka. We are moving from 0.10.0 to 2.0.0 and there have been significant changes