[
https://issues.apache.org/jira/browse/STORM-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993804#comment-14993804
]
Robert Joseph Evans commented on STORM-1179:
--------------------------------------------
Perhaps we need to define better what types of tests we want and how do we want
to run them.
I see several different levels/types of testing that should be run at different
frequencies. All of these are things that we need to do, but may not all fully
be a part of the community for various reasons.
1) True unit tests. They run fast enough that they should not be a burden to
have on by default and be a part of any build. They should frequently mock out
dependencies and try to concentrate on only executing/testing a relatively
small defined piece of code.
2) LocalCluster integration tests. These are tests that put all of the parts
of storm-core together and validate that it works as a system. Some things
that are hard to test on their own may need to be in here too. Currently way
too many of our tests are like this. These should be able to test various
configuration scenarios for correctness. I personally feel that these don't
need to run as part of the build by default, but should be run as a part of CI.
3) Cluster Integration tests. This is where ductape comes in. These should
spin up not just storm, but the other components that storm can talk to.
H-Base, HDFS, Redis, Elastisearch, Kerberos, etc. These don't need to be
exhaustive tests trying lots of different configurations, as spinning up a
cluster and running a topology can be rather slow. These should mostly test
that everything can still talk to each other and we can get the expected
output. I would love to see these as a part of CI, but I don't see travis-ci
supporting it.
4) Failure/Stress tests. These should ideally use ductape too. They should
take a set of topologies and run through various situations, especially around
trident, where we kill things, wait for recovery and verify we didn't do the
wrong thing. These can be extended in the future to include slow network, etc
as we begin to support mitigations for these situations. These too I would
love to to see as a part of CI, but again I don't see travis CI supporting it.
5) Performance tests, probably using some of ductape to set things up but maybe
not. These I would see as an extended version of ThroughputVsLatency. Ideally
we could have one that does trident and one that can work with acking disabled,
and perhaps even a more real world test that interacts with other systems.
These are looking at what the maximum throughput is that we can support for a
given latency. These are things that are not simple to automatically evaluate,
and really should be run on the same hardware each time so that results can be
truly comparable. Ideally the code is in open source, but actually running
them may have to be done outside of the community CI.
6) Scale tests. How large of a cluster can we support and how large of an
individual topology can we support. This is probably similar to the
performance tests, but running at a much larger scale. We should be looking at
the performance of topologies, and various operations, specifically nimbus and
UI for a very large cluster. I would love to see some/all of the code for this
be in open source, but running it regularly will probably have to be done
outside of the community because of a lack of hardware, and the need for manual
evaluation of the results.
I think we should concentrate on the first three, with the first two covered by
this JIRA, and the third covered by a separate JIRA. We can then branch off
into automating the others once the first three are done, and we have developed
experience with ductape, or whatever system we decide to use.
> Create Maven Profiles for Integration Tests
> -------------------------------------------
>
> Key: STORM-1179
> URL: https://issues.apache.org/jira/browse/STORM-1179
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
> Affects Versions: 0.11.0
> Reporter: Hugo Louro
> Assignee: Hugo Louro
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
