I have set it up for another open source effort in the past and it was not very hard. Am happy to volunteer if needed.
Thx Debo Sent from my iPhone > On Apr 10, 2016, at 5:53 PM, James Sirota <[email protected]> wrote: > > I’d be open to an IRC channel. Does anyone know if Apache allows this? If > yes, does anyone know how to set one up? > > Thanks, > James > > > > >> On 4/10/16, 4:52 PM, "Debojyoti Dutta" <[email protected]> wrote: >> >> Hi Nick >> >> I like your suggestions. For the enrichment layer do you think it would also >> include any advanced analytics. Else we might want to have an analytics >> layer. >> >> It would be good to have an arch which could be extended for new >> functionality. >> >> However Ryan's suggestion of the ui API and deployer also makes sense. >> >> Should we have an IRC channel to discuss this or maybe etherpad? >> >> Debo >> >> Sent from my iPhone >> >>> On Apr 10, 2016, at 4:36 PM, Nick Allen <[email protected]> wrote: >>> >>> It might help to think of our code base as four separate types of >>> functionality. This is primarily meant to give us a framework to think >>> about the organization of Metron (and drive more discussion), rather than >>> my proposal for a specific structure. >>> >>> - Sensor - Anything that captures external, non-streaming data and >>> presents it in a form ready for stream processing. >>> - Input - Responsible for preparing streaming data for enrichment. The >>> existing "parsers" fit neatly into this space. >>> - Enrichment - Responsible for enriching an incoming data feed like >>> geoip, asset enrichment, threat intel lookups, etc. >>> - Output - Responsible for persisting data that has been processed by >>> Metron which obviously means search indexers or data stores. >>> >>> >>> >>> >>> >>> On Fri, Apr 8, 2016 at 4:46 PM, Ryan Merriman <[email protected]> >>> wrote: >>> >>>> All, >>>> >>>> I would like to propose a review and refactor of the current project >>>> organization within Metron. Much of the way the legacy code was organized >>>> does not make sense anymore and could be designed so that it is easier to >>>> navigate and understand. Our test coverage has increased substantially so >>>> I believe we can do this with confidence. >>>> >>>> First off, I think we should agree on a naming convention. I see some >>>> projects (YARN and Storm for example) that prepend the sub-project with the >>>> name of the top-level project (storm-core for example). Metron also >>>> currently does this (Metron-Common). I think that's fine, although in the >>>> case of Metron, I feel like having "Metron" prepended is redundant. >>>> Regardless of whether we decide to stick with that approach, I propose that >>>> project names be uniform and lowercase. For example, under these >>>> assumptions "Metron-Common" would change to "common". >>>> >>>> The first level of organization makes sense to me. Only change I would >>>> make would be to project names: >>>> >>>> * deployment >>>> * streaming >>>> * ui >>>> >>>> Or if we want to keep metron in project names: >>>> >>>> * metron-deployment >>>> * metron-streaming >>>> * metron-ui >>>> >>>> For now I don't see any changes necessary in deployment or ui >>>> organization. I see the streaming project structure primarily driven by 2 >>>> things: the Maven dependency tree and deployment targets. For example, >>>> solr and elasticsearch code should be separated (because their dependency >>>> on lucene conflicts) but both will depend on common enrichment code. Also, >>>> now that parser, enrichment and pcap topologies are separate, code for >>>> those topologies will be deployed as separate jars. No reason to include >>>> parser code in enrichment topologies and vice-versa. Any other >>>> considerations I'm missing? >>>> >>>> With that being said, here is my initial proposal: >>>> >>>> * common - Any common code that all topologies depend on >>>> (configuration classes, generic writers for example). No dependencies on >>>> other Metron projects. >>>> * test - Contains utilities for writing unit tests, sample configs and >>>> sample data. Will depend on common. >>>> * integration-test - Contains utilities and classes needed to run our >>>> integration tests (in memory components for example). Will depend on >>>> common and test. >>>> * dataload - Contains all code related to data loading. Will also >>>> include any property files needed and integration tests. Will depend on >>>> common, test (test scope), and integration-test (test scope). >>>> * parser - All code specific to the parser topologies. Would also >>>> include scripts, property files, flux files and parser topology integration >>>> tests. This project will depend on common, test (test scope), and >>>> integration-testing (test scope). >>>> * enrichment - All code specific to the enrichment topologies (except >>>> solr and elasticsearch). Would also include scripts, property files, flux >>>> files and enrichment topology integration tests. This project will depend >>>> on common, test (test scope), and integration-test (test scope). >>>> * elasticsearch - All Elasticsearch related code. Will depend on >>>> enrichment. >>>> * solr - All Solr related code. Will depend on enrichment. >>>> * pcap - All code specific to the topology dedicated to pcap. Would >>>> also include scripts, property files, flux files and pcap integration >>>> test. This project will depend on common, test (test scope) and >>>> integration-test (test scope). >>>> * api - This will serve as a generic replacement for >>>> Metron-Pcap_Service. Will contain all code to build a Metron web service >>>> middle layer that can expose APIs through REST or other client protocols. >>>> Could possibly depend on all other projects or separated further if version >>>> conflicts arise (separate api projects for solr and elasticsearch for >>>> example). >>>> >>>> Looking forward to hearing everyone's feedback and great ideas. >>>> >>>> Ryan Merriman >>> >>> >>> >>> -- >>> Nick Allen <[email protected]> >>
