Hi James Ok set it up and ack …..
Thx On 4/10/16, 6:31 PM, "James Sirota" <[email protected]> wrote: >Hi Debo, > >I think it would be great if you set it up > >Thanks, >James > > > > >On 4/10/16, 6:25 PM, "Debojyoti Dutta" <[email protected]> wrote: > >>I have set it up for another open source effort in the past and it was not >>very hard. Am happy to volunteer if needed. >> >>Thx >>Debo >> >>Sent from my iPhone >> >>> On Apr 10, 2016, at 5:53 PM, James Sirota <[email protected]> wrote: >>> >>> I’d be open to an IRC channel. Does anyone know if Apache allows this? If >>> yes, does anyone know how to set one up? >>> >>> Thanks, >>> James >>> >>> >>> >>> >>>> On 4/10/16, 4:52 PM, "Debojyoti Dutta" <[email protected]> wrote: >>>> >>>> Hi Nick >>>> >>>> I like your suggestions. For the enrichment layer do you think it would >>>> also include any advanced analytics. Else we might want to have an >>>> analytics layer. >>>> >>>> It would be good to have an arch which could be extended for new >>>> functionality. >>>> >>>> However Ryan's suggestion of the ui API and deployer also makes sense. >>>> >>>> Should we have an IRC channel to discuss this or maybe etherpad? >>>> >>>> Debo >>>> >>>> Sent from my iPhone >>>> >>>>> On Apr 10, 2016, at 4:36 PM, Nick Allen <[email protected]> wrote: >>>>> >>>>> It might help to think of our code base as four separate types of >>>>> functionality. This is primarily meant to give us a framework to think >>>>> about the organization of Metron (and drive more discussion), rather than >>>>> my proposal for a specific structure. >>>>> >>>>> - Sensor - Anything that captures external, non-streaming data and >>>>> presents it in a form ready for stream processing. >>>>> - Input - Responsible for preparing streaming data for enrichment. The >>>>> existing "parsers" fit neatly into this space. >>>>> - Enrichment - Responsible for enriching an incoming data feed like >>>>> geoip, asset enrichment, threat intel lookups, etc. >>>>> - Output - Responsible for persisting data that has been processed by >>>>> Metron which obviously means search indexers or data stores. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Apr 8, 2016 at 4:46 PM, Ryan Merriman <[email protected]> >>>>> wrote: >>>>> >>>>>> All, >>>>>> >>>>>> I would like to propose a review and refactor of the current project >>>>>> organization within Metron. Much of the way the legacy code was >>>>>> organized >>>>>> does not make sense anymore and could be designed so that it is easier to >>>>>> navigate and understand. Our test coverage has increased substantially >>>>>> so >>>>>> I believe we can do this with confidence. >>>>>> >>>>>> First off, I think we should agree on a naming convention. I see some >>>>>> projects (YARN and Storm for example) that prepend the sub-project with >>>>>> the >>>>>> name of the top-level project (storm-core for example). Metron also >>>>>> currently does this (Metron-Common). I think that's fine, although in >>>>>> the >>>>>> case of Metron, I feel like having "Metron" prepended is redundant. >>>>>> Regardless of whether we decide to stick with that approach, I propose >>>>>> that >>>>>> project names be uniform and lowercase. For example, under these >>>>>> assumptions "Metron-Common" would change to "common". >>>>>> >>>>>> The first level of organization makes sense to me. Only change I would >>>>>> make would be to project names: >>>>>> >>>>>> * deployment >>>>>> * streaming >>>>>> * ui >>>>>> >>>>>> Or if we want to keep metron in project names: >>>>>> >>>>>> * metron-deployment >>>>>> * metron-streaming >>>>>> * metron-ui >>>>>> >>>>>> For now I don't see any changes necessary in deployment or ui >>>>>> organization. I see the streaming project structure primarily driven by >>>>>> 2 >>>>>> things: the Maven dependency tree and deployment targets. For example, >>>>>> solr and elasticsearch code should be separated (because their dependency >>>>>> on lucene conflicts) but both will depend on common enrichment code. >>>>>> Also, >>>>>> now that parser, enrichment and pcap topologies are separate, code for >>>>>> those topologies will be deployed as separate jars. No reason to include >>>>>> parser code in enrichment topologies and vice-versa. Any other >>>>>> considerations I'm missing? >>>>>> >>>>>> With that being said, here is my initial proposal: >>>>>> >>>>>> * common - Any common code that all topologies depend on >>>>>> (configuration classes, generic writers for example). No dependencies on >>>>>> other Metron projects. >>>>>> * test - Contains utilities for writing unit tests, sample configs and >>>>>> sample data. Will depend on common. >>>>>> * integration-test - Contains utilities and classes needed to run our >>>>>> integration tests (in memory components for example). Will depend on >>>>>> common and test. >>>>>> * dataload - Contains all code related to data loading. Will also >>>>>> include any property files needed and integration tests. Will depend on >>>>>> common, test (test scope), and integration-test (test scope). >>>>>> * parser - All code specific to the parser topologies. Would also >>>>>> include scripts, property files, flux files and parser topology >>>>>> integration >>>>>> tests. This project will depend on common, test (test scope), and >>>>>> integration-testing (test scope). >>>>>> * enrichment - All code specific to the enrichment topologies (except >>>>>> solr and elasticsearch). Would also include scripts, property files, >>>>>> flux >>>>>> files and enrichment topology integration tests. This project will >>>>>> depend >>>>>> on common, test (test scope), and integration-test (test scope). >>>>>> * elasticsearch - All Elasticsearch related code. Will depend on >>>>>> enrichment. >>>>>> * solr - All Solr related code. Will depend on enrichment. >>>>>> * pcap - All code specific to the topology dedicated to pcap. Would >>>>>> also include scripts, property files, flux files and pcap integration >>>>>> test. This project will depend on common, test (test scope) and >>>>>> integration-test (test scope). >>>>>> * api - This will serve as a generic replacement for >>>>>> Metron-Pcap_Service. Will contain all code to build a Metron web service >>>>>> middle layer that can expose APIs through REST or other client protocols. >>>>>> Could possibly depend on all other projects or separated further if >>>>>> version >>>>>> conflicts arise (separate api projects for solr and elasticsearch for >>>>>> example). >>>>>> >>>>>> Looking forward to hearing everyone's feedback and great ideas. >>>>>> >>>>>> Ryan Merriman >>>>> >>>>> >>>>> >>>>> -- >>>>> Nick Allen <[email protected]> >>>> >>
