Re: [DISCUSS] Lowering the barrier to entry to for new users

2017-12-21 Thread Justin Leet
I just opened https://github.com/apache/metron/pull/881, for a CONTRIBUTING.md after remembering I'd done two-thirds of it forever ago. Feel free to provide feedback on the PR. On Wed, Dec 20, 2017 at 5:49 AM, zeo...@gmail.com wrote: > I agree we should streamline #2 and lower

[GitHub] metron pull request #881: METRON-1071: Create CONTRIBUTING.md

2017-12-21 Thread justinleet
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/881 METRON-1071: Create CONTRIBUTING.md ## Contributor Comments Totally forgot I'd been sitting on a half done version of this. It's basically a stripped down version of the Development

[GitHub] metron pull request #880: METRON-1379: Add an OBJECT_GET stellar function

2017-12-21 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/880 METRON-1379: Add an OBJECT_GET stellar function ## Contributor Comments With the creation of METRON-1378 we have the ability to create serialized summaries of data. We need to interact

[GitHub] metron pull request #871: METRON-1365: Allow PROFILE_GET to return a default...

2017-12-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/871 ---

[GitHub] metron pull request #879: METRON-1378: Create a summarizer

2017-12-21 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/879 METRON-1378: Create a summarizer ## Contributor Comments We have a nice and generalized infrastructure for loading data into HBase and interacting with it via `flatfile_loader.sh` and

Re: Metron nested object

2017-12-21 Thread Simon Elliston Ball
Correct, nested objects in lucene indexes lead to sub-documents, which leads to a massive drop in ingest and query rates, this is why the JSONMap parser for example deliberately flattens the Metorn JSON object. Before this decision was made, very early versions of OpenSOC nested enrichments for

Re: Metron nested object

2017-12-21 Thread Laurens Vets
Hi Ali, We have recently faced some data sources that generate data in a nested format. For example, AWS Cloudtrail generates data in the following JSON format: { "Records": [ { "eventVersion": *"2.0"*, "userIdentity": { "type": *"IAMUser"*, "principalId":

Re: Secure code analysis

2017-12-21 Thread zeo...@gmail.com
Just following up on this conversation again - I have discussed this ad-hoc with a few PMC members recently and wanted to bring it up on the list. Veracode has provided us with a 100% free portal to scan the Metron code with, but in order to integrate, the safest option is probably to use the

[GitHub] metron pull request #864: METRON-1348 Metron Service Checks Use Wrong Hostna...

2017-12-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/864 ---

[GitHub] metron issue #864: METRON-1348 Metron Service Checks Use Wrong Hostname

2017-12-21 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/864 I appreciate the reviews @ottobackwards and @anandsubbu . ---

Re: Metron nested object

2017-12-21 Thread Ali Nazemian
So Metron enrichment and indexer are not nested aware? Is there any plan to add that to Metron in future? Cheers, Ali On Fri, Dec 22, 2017 at 12:46 AM, Otto Fowler wrote: > I believe right now you have to flatten. > The jsonMap parser does this. > > > On December 21,

Re: Metron nested object

2017-12-21 Thread Otto Fowler
I believe right now you have to flatten. The jsonMap parser does this. On December 21, 2017 at 08:28:13, Ali Nazemian (alinazem...@gmail.com) wrote: Hi all, We have recently faced some data sources that generate data in a nested format. For example, AWS Cloudtrail generates data in the

Metron nested object

2017-12-21 Thread Ali Nazemian
Hi all, We have recently faced some data sources that generate data in a nested format. For example, AWS Cloudtrail generates data in the following JSON format: { "Records": [ { "eventVersion": *"2.0"*, "userIdentity": { "type": *"IAMUser"*,