+1 to option 3 on both. Also strongly in favor of Docker. We recently took a similar approach in metron-bro-plugin-kafka as well (link <https://github.com/apache/metron-bro-plugin-kafka/tree/master/docker>) to do end to end testing.
Jon On Fri, Mar 8, 2019 at 9:53 AM Nick Allen <n...@nickallen.org> wrote: > +1 for option 3. I am in favor of using Docker for the integration tests > for all the reasons that you mentioned. > > On Fri, Mar 8, 2019 at 9:47 AM Ryan Merriman <merrim...@gmail.com> wrote: > > > I have been researching the effort involved to upgrade to HDP 3. Along > the > > way I've found a couple challenging issues that we will need to solve, > both > > involving our integration testing strategy. > > > > The first issue is Kafka. We are moving from 0.10.0 to 2.0.0 and there > > have been significant changes to the API. This creates an issue in the > > KafkaComponent class, which we use as an in-memory Kafka server in > > integration tests. Most of the classes that were previously used have > gone > > away, and to the best of my knowledge, were not supported as public APIs. > > I also don't see any publicly documented APIs to replace them. > > > > The second issue is HBase. We are moving from 1.1.2 to 2.0.2 so another > > significant change. This creates an issue in the MockHTable class > > becausethe HTableInterface class has changed to Table, essentially > > requiring that MockHTable be rewritten to conform to the new interface. > > It's my opinion that this class is complicated and difficult to maintain > as > > it is anyways. > > > > These 2 issues have the potential to add a significant amount of work to > > upgrading Metron to HDP 3. I want to take a step back and review our > > options before we move forward. Here are some initial thoughts I had on > > how to approach this. For HBase: > > > > 1. Update MockHTable to work with the new HBase API. We would > continue > > using a mock server approach for HBase. > > 2. Research replacing MockHTable with an in-memory HBase server. > > 3. Replace MockHTable with a Docker container running HBase. > > > > For Kafka: > > > > 1. Replace KafkaComponent with a mock server implementation. > > 2. Update KafkaComponent to work with the new API. We would probably > > need to leverage some internal Kafka classes. I do not see a testing > > API > > documented publicly. > > 3. Replace KafkaComponent with a Docker container running Kafka. > > > > What other options are there? Whatever we choose I think we should > follow > > a similar approach for both (mock servers, in memory servers, Docker, > other > > options I'm not thinking of). > > > > This will not shock anyone but I would be in favor of Docker containers. > > They have the advantage of classpath isolation, easy upgrades, and > accurate > > integration testing. The downside is we will have to adjusts our tests > and > > travis script to incorporate these Docker containers into our build > > process. We have discussed this at length in the past and it has > generally > > stalled for various reasons. Maybe if we move a few services at a time > it > > might be more palatable? As for the other 2 approaches, I think if > either > > worked well we wouldn't be having this discussion. Mock servers are hard > > to maintain and I don't see in memory testing classes documented in > > javadocs for either service. > > > > Thoughts? > > > -- Jon Zeolla
