Re: Metron nested object

2018-01-11 Thread Simon Elliston Ball
I’m all for adding extra stores, especially once we have separated indexing topologies. Druid (and therefore a ui based on superset) seems an obvious logical store to me. That said, the scheme management starts to feel like it needs some thought once we have enough range of schema sensitive

Re: Metron nested object

2018-01-11 Thread Andre
Simon, With the risk of sounding like an heretic: Is there any particular reason Metron still considers ES as the "default"[1] fast access data store? Sometimes I wonder if we wouldn't be better off leveraging schema evolution friendly formats with UIs like SuperSets? Probably not as fast as

Re: Metron nested object

2017-12-21 Thread Simon Elliston Ball
Correct, nested objects in lucene indexes lead to sub-documents, which leads to a massive drop in ingest and query rates, this is why the JSONMap parser for example deliberately flattens the Metorn JSON object. Before this decision was made, very early versions of OpenSOC nested enrichments for

Re: Metron nested object

2017-12-21 Thread Laurens Vets
Hi Ali, We have recently faced some data sources that generate data in a nested format. For example, AWS Cloudtrail generates data in the following JSON format: { "Records": [ { "eventVersion": *"2.0"*, "userIdentity": { "type": *"IAMUser"*, "principalId":

Re: Metron nested object

2017-12-21 Thread Ali Nazemian
So Metron enrichment and indexer are not nested aware? Is there any plan to add that to Metron in future? Cheers, Ali On Fri, Dec 22, 2017 at 12:46 AM, Otto Fowler wrote: > I believe right now you have to flatten. > The jsonMap parser does this. > > > On December 21,

Re: Metron nested object

2017-12-21 Thread Otto Fowler
I believe right now you have to flatten. The jsonMap parser does this. On December 21, 2017 at 08:28:13, Ali Nazemian (alinazem...@gmail.com) wrote: Hi all, We have recently faced some data sources that generate data in a nested format. For example, AWS Cloudtrail generates data in the

Metron nested object

2017-12-21 Thread Ali Nazemian
Hi all, We have recently faced some data sources that generate data in a nested format. For example, AWS Cloudtrail generates data in the following JSON format: { "Records": [ { "eventVersion": *"2.0"*, "userIdentity": { "type": *"IAMUser"*,