Jon, I didn't get a chance to review your PR, but it immediately came to mind wrt this task - did you guys have that working in Travis with Maven? What's the container lifecycle you came up with look like?
On Fri, Mar 8, 2019 at 8:30 AM zeo...@gmail.com <zeo...@gmail.com> wrote: > +1 to option 3 on both. Also strongly in favor of Docker. We recently > took a similar approach in metron-bro-plugin-kafka as well (link > <https://github.com/apache/metron-bro-plugin-kafka/tree/master/docker>) to > do end to end testing. > > Jon > > On Fri, Mar 8, 2019 at 9:53 AM Nick Allen <n...@nickallen.org> wrote: > > > +1 for option 3. I am in favor of using Docker for the integration tests > > for all the reasons that you mentioned. > > > > On Fri, Mar 8, 2019 at 9:47 AM Ryan Merriman <merrim...@gmail.com> > wrote: > > > > > I have been researching the effort involved to upgrade to HDP 3. Along > > the > > > way I've found a couple challenging issues that we will need to solve, > > both > > > involving our integration testing strategy. > > > > > > The first issue is Kafka. We are moving from 0.10.0 to 2.0.0 and there > > > have been significant changes to the API. This creates an issue in the > > > KafkaComponent class, which we use as an in-memory Kafka server in > > > integration tests. Most of the classes that were previously used have > > gone > > > away, and to the best of my knowledge, were not supported as public > APIs. > > > I also don't see any publicly documented APIs to replace them. > > > > > > The second issue is HBase. We are moving from 1.1.2 to 2.0.2 so > another > > > significant change. This creates an issue in the MockHTable class > > > becausethe HTableInterface class has changed to Table, essentially > > > requiring that MockHTable be rewritten to conform to the new interface. > > > It's my opinion that this class is complicated and difficult to > maintain > > as > > > it is anyways. > > > > > > These 2 issues have the potential to add a significant amount of work > to > > > upgrading Metron to HDP 3. I want to take a step back and review our > > > options before we move forward. Here are some initial thoughts I had > on > > > how to approach this. For HBase: > > > > > > 1. Update MockHTable to work with the new HBase API. We would > > continue > > > using a mock server approach for HBase. > > > 2. Research replacing MockHTable with an in-memory HBase server. > > > 3. Replace MockHTable with a Docker container running HBase. > > > > > > For Kafka: > > > > > > 1. Replace KafkaComponent with a mock server implementation. > > > 2. Update KafkaComponent to work with the new API. We would > probably > > > need to leverage some internal Kafka classes. I do not see a > testing > > > API > > > documented publicly. > > > 3. Replace KafkaComponent with a Docker container running Kafka. > > > > > > What other options are there? Whatever we choose I think we should > > follow > > > a similar approach for both (mock servers, in memory servers, Docker, > > other > > > options I'm not thinking of). > > > > > > This will not shock anyone but I would be in favor of Docker > containers. > > > They have the advantage of classpath isolation, easy upgrades, and > > accurate > > > integration testing. The downside is we will have to adjusts our tests > > and > > > travis script to incorporate these Docker containers into our build > > > process. We have discussed this at length in the past and it has > > generally > > > stalled for various reasons. Maybe if we move a few services at a time > > it > > > might be more palatable? As for the other 2 approaches, I think if > > either > > > worked well we wouldn't be having this discussion. Mock servers are > hard > > > to maintain and I don't see in memory testing classes documented in > > > javadocs for either service. > > > > > > Thoughts? > > > > > > -- > > Jon Zeolla >
