I also like Mans categories, as I know I have personally written some unit tests that don't require any external dependencies, but might have veered closer to an integration test than a true unit test. Some examples I am thinking of are the tests for Put/GetSolr which use and EmbeddedSolrServer, or TestListenSyslog which actually starts listening on a port and receives messages over a socket. I would still want those tests to run as part of the normal build, which I think is what Mans is suggesting.
I'm not sure if the "no external dependencies" would always hold true as a rule though. I could see there being some more complex tests between internal components that might not have external dependencies, but still makes sense to have in the IT part of the build. As an example, if we ever create a mechanism to connect two processors together and test the flow between them, this feels more like a true integration test, yet it might not have any external dependencies. So maybe it is a case by case basis. On Wed, Jan 13, 2016 at 3:02 PM, Joe Skora <jsk...@gmail.com> wrote: > I like that Mans' categories. > > Logically, I think the 2nd and 3rd categories fold together with "no > dependency" integration tests being another case alongside "hadoop > dependency", "jms dependency", "aws dependency", etc. > > Then I would expect that > > - every component should have unit tests, > - most components (if not every) should have "no dependency" integration > tests, and > - externally connected components should have "external dependency" > integration tests. > > > > On Wed, Jan 13, 2016 at 2:39 PM, M Singh <mans2si...@yahoo.com.invalid> > wrote: > > > Hi: > > My thought is that we can classify tests into 3 major categories - > > - Pure unit test - class isolated using mocks etc > > - Integration with no external dependencies - some interaction with > > other classes/in memory db, or mocks, but no external resources. > > - Integration with external resource upto and including end-to-end - > > Which require real external resources (http endpoint, aws etc) > > > > In this way, we can always keep unit and integration w/o ext dependencies > > tests while conditionally enable integration tests. > > Mans > > > > On Wednesday, January 13, 2016 11:24 AM, Aldrin Piri < > > aldrinp...@gmail.com> wrote: > > > > > > Definitely agree and can see value in those as well. The core issue I > am > > trying to address with this is that as highlighted by Joe. > > > > As an intermediate step, I have created a PR [1] that shows how I > envision > > a baseline of this. Currently this treats all ITs (in this external > > resource context) the same without separate profiles. What I am trying > to > > do is not impede a place for unit tests (or, perhaps more generically, > > those that can run as part of the build). > > > > [1] https://github.com/apache/nifi/pull/173 > > > > On Wed, Jan 13, 2016 at 8:19 AM, Oleg Zhurakousky < > > ozhurakou...@hortonworks.com> wrote: > > > > > Aldrin > > > > > > IMHO there are two types of integration testing; 1. Integration > testing a > > > Processor/ControllerService with the actual target system, 2. > Integration > > > testing of the flow or part of the flow. > > > The second one essentially is the same as the first one but the target > > > system is being NiFi itself and we are seriously lacking on that type > of > > > testing since NiFi is highly modularized and there is not a single > module > > > where such testing could be performed. As I’ve mentioned earlier in > this > > > list, I’ve already started such module on my fork > > > https://github.com/olegz/nifi/tree/int-test which allowed me already > to > > > discover a few bugs and I think we need to start talking about pushing > it > > > to the trunk. > > > > > > So what I am essentially saying is that while profile and other maven > > > tricks may set a convention to allow one to provide "conditional target > > > system testing” (which is essentially what you mean by integration test > > > with AWS etc), we still need to think about automating internal NiFi > > > integration testing. > > > > > > Cheers > > > Oleg > > > > > > > > > On Jan 12, 2016, at 10:26 PM, Aldrin Piri <aldrinp...@gmail.com > <mailto: > > > aldrinp...@gmail.com>> wrote: > > > > > > All, > > > > > > We have had a few conversations regarding integration tests in the past > > In > > > part, this issue has resurfaced thanks to the work going on with > > NIFI-1325 > > > in extending AWS related processors and extensions. The issue at hand > is > > > that, In this particular case, the contributor made test cases to > > > supplement their contribution but, because the integration level test > > > classes are ignored, had to create separate unit test classes to get > > these > > > tests to run. AbstractS3Test lays the ground work quite appropriately > > for > > > integration testing, but doesn't so much apply to unit tests where > > > possible. > > > > > > What I am interested in is establishing direction for how these > > integration > > > level tests should work within our codebase; superficially, what are > the > > > conventions that make sense and how do we treat these within our > > codebase. > > > There is certainly grey area between unit and integration level tests > at > > > spots within our codebase, but for the immediate future, I am looking > to > > > establish a basis in terms of conventions so that we can avoid similar > > > issues moving forward and perhaps get some guidelines that the > community > > > can use to provide these. > > > > > > Following the pattern that is present in support of using the > > > maven-surefire-plugin, I could see a similar pattern of IT<Component> > to > > > complement the Test<Component>. This is a default pattern that is > > > supported by the maven-failsafe-plugin, a likely candidate to drive > this > > in > > > our build process under some associated profile [1]. > > > > > > In conjunction with this, I would like to see a profile that defaults > to > > > being deactivated that runs integration level tests and migrate the > > > integration tests (AbstractS3Test and its subclasses) to an agreed upon > > > convention as previously mentioned. > > > > > > We have an issue associated with this unit/integration level isolation, > > > NIFI-569 [2]. > > > > > > Any thoughts toward this would be most appreciated and I shall look to > > > incorporate the points made in the associated issues in JIRA. > > > > > > Thanks! > > > > > > [1] > > > > > > > > > http://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html > > > [2] https://issues.apache.org/jira/browse/NIFI-569 > > > > > > > > > > > > >
