I completely missed this email the first time around. I would be happy to help as well - actually as a part of testing some changes to the bro parts in Metron I have been using a few public datasets, and will be adding some into the bro tests for logs other than dns and http.
Jon On Thu, May 4, 2017, 5:43 AM Dima Kovalyov <[email protected]> wrote: > Hello Matt, > > It's been long-time for us to continue working in this direction further. > Thank you for the response. > > I wanted to ask if anything changed since our last discussion regarding > parsers, enrichments and generators contribution. Is there anything else we > should be doing other then: > 1. Sign Corporate CLA with Apache (link).< > https://www.apache.org/licenses/#clas> > 2. Sign an Individual CLA for the submitter (instructions< > https://www.apache.org/licenses/#clas>), I need to do that despite #1? > 3. Register on Apache GitHub and JIRA. > 4. Open JIRA master ticket for submissions from SSTECH. > 5. Create sub-task for each piece of code we are going to submit. > 6. Send email to [email protected]<mailto:[email protected]> > describing proposed changes including JIRA case. What to expect from email? > Approval or suggestions? > 7. Fork Apache Metron master branch internally, merge our changes and test > them using single-node vagrant. > 8. Create Pull Request (PR), how? > 9. Wait for the dev team to review, accept changes and answer any > questions or suggestions. > > This above applies to the code that was: > 1. Written and tested. > 2. Covered with unit tests. > 3. Can be built using maven > 4. Has place in the Apache Metron folder tree. > > - Dima > > > On 10/08/2016 06:43 AM, Matt Foley wrote: > Hi Dima, > Sorry this is getting a little long, but TL;DR on > Metron+Development+Environment+Setup+Instructions< > https://cwiki.apache.org/confluence/display/METRON/Metron+Development+Environment+Setup+Instructions> > is: > > A. Open a Jira for the work you want to do, or the contribution you want > to make. Since you have several parsers, you might open an umbrella Jira, > with four subtask jiras, each of which includes the parser and test data > generator for one of the four technologies you mentioned. > B. Send an email to the dev list proposing what you want to submit, and > referencing the Jira. > C. Fork the Apache Metron code base in your personal github area. > D. Make sure your contribution works correctly with the latest master > branch code. > E. Decide where in the code tree your contribution would fit best. The > parsers themselves would of course go under > metron-platform/metron-parsers/. The data generators could reasonably be > put in the test/ subdirectory, perhaps under > metron-platform/metron-parsers/src/test/java/org/apache/metron/writers > (although we would defer to the reviewers). > F. Add the necessary maven glue so the new pieces build along with the > core. > G. Metron requires all submissions to have unit tests with thorough > coverage, so add those if they aren’t there yet. > H. When things are ready to submit, commit everything to your github, and > create a Pull Request (PR) > I. Watch the PR and Jira for responses. Respond to questions, accept > feedback or suggest alternative solutions, and work through the process > with the community. If things need lengthy discussion, you may be asked to > do so in the dev list. > J. With patience, all issues will be agreed on, and the contribution will > be accepted into Metron, for the benefit of the whole community. > > Hope this helps. Feel free to contact me directly, or just ask questions > on the dev list. > Best regards, > —Matt > > > On Oct 7, 2016, at 6:05 PM, Matt Foley <[email protected]<mailto: > [email protected]>> wrote: > > Dima, that’s great! > > Since you’re talking about a code contribution (or several :-), let’s move > the discussion over to the [email protected]<mailto: > [email protected]> list, after this response. Briefly, > here’s how you submit a contribution. > > First the housekeeping: > 1. If Sstech has not yet signed a Corporate CLA with Apache, please ask > them to do so (instructions<https://www.apache.org/licenses/#clas>) > 2. If you, or a colleague who will submit the contributions, has not yet > signed an Individual CLA, please do so (instructions< > https://www.apache.org/licenses/#clas>) > > Since you’ve been successfully writing Metron parsers, you almost > certainly have already done the following, but I’ll mention them here for > the sake of other readers: > 3. If you’re not on the dev mailing list, please join it (instructions< > https://cwiki.apache.org/confluence/display/METRON/Community+Resources>) > 4. If you weren’t a registered user of Apache’s Jira, you would request to > be added, but I see you already are, so that’s good. > 5. If you don’t yet have an account on Github.com<http://github.com/>, > sign up for one (the free level is fine). > 6. Set up a Metron Development Environment, and establish the ability to > spin up a single-node test environment (instructions< > https://cwiki.apache.org/confluence/display/METRON/Metron+Development+Environment+Setup+Instructions > >) > > To actually make the contribution, you follow the process shown in: > > https://cwiki.apache.org/confluence/display/METRON/Metron+Development+Environment+Setup+Instructions > > I’ll go into more detail in a direct email. > Thanks a lot for being interested in submitting these! > > Cheers, > —Matt > > ________________________________ > From: Dima Kovalyov <[email protected]<mailto: > [email protected]>> > Sent: Friday, October 07, 2016 4:44 PM > To: [email protected]<mailto: > [email protected]>; Satish Abburi > Subject: Re: [CALL FOR TEST DATA] Request help identifying public domain > or opensource test data sets for Metron testing > > Hello Matt, > > We (Sstech team) currently have parsers and data generators for BlueCoat, > Unix, MS Exchange, MS Windows and we would gladly contribute them. > > Can you please share the procedure for submitting these peaces? > Thank you. > > - Dima > > On 10/08/2016 01:49 AM, Matt Foley wrote: > Hi all, > Enhanced testing of Metron, especially performance testing, would be aided > by having data sets of realistic size, that exercise one or more of the > various parts of Metron: > > * each Parser (bro, yaf, snort, squid, ...) > * each Enhancer (geo, user, assets, ...) > * each Threat Intel module (Soltra, HailATaxi, ...) > > Data sets must meet the following criteria: > > * opensource or public domain > * suitably scrubbed, containing no Personally Identifiable Information > * unencumbered by company sensitivity, security, or IP concerns. > > They may take the form of raw PCAP streams, or they may be already parsed > or otherwise pre-processed. > > If you know of opensource or public domain data sets of this kind, please > respond with the URL, in this email thread or to the Jira ticket METRON-491< > https://issues.apache.org/jira/browse/METRON-491>. > > If you have an appropriate data set that your company would be willing to > contribute, please also respond and we will help in any way we can. > > > Thanks, > --Matt > > > > -- Jon
