Is there any reason to keep the "test" and "integration-test" code separate?
On Fri, Apr 8, 2016 at 4:46 PM, Ryan Merriman <[email protected]> wrote: > All, > > I would like to propose a review and refactor of the current project > organization within Metron. Much of the way the legacy code was organized > does not make sense anymore and could be designed so that it is easier to > navigate and understand. Our test coverage has increased substantially so > I believe we can do this with confidence. > > First off, I think we should agree on a naming convention. I see some > projects (YARN and Storm for example) that prepend the sub-project with the > name of the top-level project (storm-core for example). Metron also > currently does this (Metron-Common). I think that's fine, although in the > case of Metron, I feel like having "Metron" prepended is redundant. > Regardless of whether we decide to stick with that approach, I propose that > project names be uniform and lowercase. For example, under these > assumptions "Metron-Common" would change to "common". > > The first level of organization makes sense to me. Only change I would > make would be to project names: > > * deployment > * streaming > * ui > > Or if we want to keep metron in project names: > > * metron-deployment > * metron-streaming > * metron-ui > > For now I don't see any changes necessary in deployment or ui > organization. I see the streaming project structure primarily driven by 2 > things: the Maven dependency tree and deployment targets. For example, > solr and elasticsearch code should be separated (because their dependency > on lucene conflicts) but both will depend on common enrichment code. Also, > now that parser, enrichment and pcap topologies are separate, code for > those topologies will be deployed as separate jars. No reason to include > parser code in enrichment topologies and vice-versa. Any other > considerations I'm missing? > > With that being said, here is my initial proposal: > > * common - Any common code that all topologies depend on > (configuration classes, generic writers for example). No dependencies on > other Metron projects. > * test - Contains utilities for writing unit tests, sample configs and > sample data. Will depend on common. > * integration-test - Contains utilities and classes needed to run our > integration tests (in memory components for example). Will depend on > common and test. > * dataload - Contains all code related to data loading. Will also > include any property files needed and integration tests. Will depend on > common, test (test scope), and integration-test (test scope). > * parser - All code specific to the parser topologies. Would also > include scripts, property files, flux files and parser topology integration > tests. This project will depend on common, test (test scope), and > integration-testing (test scope). > * enrichment - All code specific to the enrichment topologies (except > solr and elasticsearch). Would also include scripts, property files, flux > files and enrichment topology integration tests. This project will depend > on common, test (test scope), and integration-test (test scope). > * elasticsearch - All Elasticsearch related code. Will depend on > enrichment. > * solr - All Solr related code. Will depend on enrichment. > * pcap - All code specific to the topology dedicated to pcap. Would > also include scripts, property files, flux files and pcap integration > test. This project will depend on common, test (test scope) and > integration-test (test scope). > * api - This will serve as a generic replacement for > Metron-Pcap_Service. Will contain all code to build a Metron web service > middle layer that can expose APIs through REST or other client protocols. > Could possibly depend on all other projects or separated further if version > conflicts arise (separate api projects for solr and elasticsearch for > example). > > Looking forward to hearing everyone's feedback and great ideas. > > Ryan Merriman > -- Nick Allen <[email protected]>
