Re: the integration testing point above, a strategy I used recently to
alleviate a similar problem was to exploit JUnit 5's extensions. I haven't
played with this at all in Metron's code, so 1) Take this with a several
grains of salt and 2) Feel encouraged to point out anything
wrong/problematic/ludicrous. My use case was pretty tame and easy to
implement.
Essentially, you can set up an extension backed by a singleton cluster (in
the case I was doing, I had two extensions, one for a Kafka cluster and one
for MiniDfs). The backing clusters expose some methods (i.e. create topic,
get brokers, get file system, etc.), which lets the tests classes setup
whatever they need. Classes that need them just use an annotation and can
be setup under the hood to init only when tests in the current run suite
actually use them and spin down after all are done. This saved ~1 min on a
4 minute build. It ends up being a pretty clean way to use them, although
we might need to be a bit more sophisticated, since my case was less
complicated. The main messiness is that this necessarily invites tests to
interfere with each other (i.e. if multiple tests use the kafka topic
"test", problems will be involved). We might be able to improve on this.
We could likely do something similar with our InMemoryComponents. Right now
these are spun up on a per-test basis, rather than the overall suite. This
involves some setup in any class that wants them, just to be able to get a
cluster. There's also some definition of spin up order and so on, which is
already taken care of with the extensions (declaration order). The other
catch in our case is that we have way more code covered by JUnit 4, which
doesn't have the same capabilities. That migration doesn't necessarily
have to be done all at once, but it is a potential substantial pain point.
Basically, I'd hope to push management of our abstraction further back into
actual JUnit so they're get more reuse across runs and it's easily
understood what a test needs and uses right up front. In our case, the
InMemoryComponents likely map to extensions.
If we did something like this, it potentially solves a few problems
* Makes it easy to build an integration test that says "Give me these
components".
* Keeps it alive across any test that needs them
* Only spins it up if there are tests running that do need them. Only spins
up the necessary components.
* Lower our build times.
* Interaction with components remains primarily through the same methods
you'd normally interact (you can build producers/consumers, or whatever).
* IMO, easier to explain and have people use. I've had a couple people
pretty easily pick up on it.
I can't share the code, but essentially it looks like (And I'm sure email
will butcher this, so I'm sorry in advance):
@ExtendWith({<ExtensionOne.class, ExtensionTwo.class, ...})
...
@BeforeAll
public static void beforeAll() {
// Test specific setup, similar to before. But without the cluster
creation work.
}
Everything else gets handled under the covers, with the caveat that what I
needed to do was less involved than some of our tests, so there may be some
experimenting.
On Wed, May 1, 2019 at 10:59 AM Justin Leet <justinjl...@gmail.com> wrote:
> Hi all,
>
> I wanted to start a discussion on something near and dear to all of our
> hearts: The role of full-dev in our testing cycle.
>
> Right now, we require all PRs to have spun up the full-dev environment and
> make sure that things flow through properly. In some cases, this is a
> necessity, and in other cases, it's a fairly large burden on current and
> potential contributors.
>
> So what do we need to do to reduce our dependence on full dev and increase
> our confidence in our CI process?
>
> My initial thoughts on this encompass a few things
> * Increasing our ability to easily write automated tests. In particular, I
> think our integration testing is fairly hard and has some structural issues
> (e.g. our tests spin up components per Test class, not for the overall test
> suite being run, which also increases build times, etc.). How do we make
> this easier and have less boilerplate? I have one potential idea I'll
> reply to this thread with.
> * Unit tests in general. I'd argue that any area we thing need full-dev
> spun up to be confident in needs more testing. Does anyone have any
> opinions on which areas we have the least confidence in? Which areas of
> code are people afraid to touch?
> * Our general procedures. Should we not be requiring full-dev on all PRs,
> but maybe just asking for a justification why not (e.g. "I don't need
> full-dev for this, because I added unit tests here and here and integration
> tests for the these components?"). And then a reviewer can request a
> full-dev spin up if needed? Could we stage this rollout (e.g.
> metron-platform modules require it, but others don't, etc.?) This'll add
> more pressure onto the release phase, so maybe that involves fleshing out a
> checklist of critical path items to check.
> * Do we need to improve our docs? Publish Javadocs? After all, if docs are
> better, hopefully we'd see less issues introduced and be more confident in
> changes coming in.
> * What environment do we even want testing to occur in?
> https://github.com/apache/metron/pull/1261 has seen a lot of work, and
> getting it across the finish line may help make the dev environment less
> onerous, even if still needed.
>
