Correct, nested objects in lucene indexes lead to sub-documents, which leads to 
a massive drop in ingest and query rates, this is why the JSONMap parser for 
example deliberately flattens the Metorn JSON object. Before this decision was 
made, very early versions of OpenSOC nested enrichments for example, but 
performance became a challenge. 

Simon


> On 21 Dec 2017, at 13:57, Ali Nazemian <alinazem...@gmail.com> wrote:
> 
> So Metron enrichment and indexer are not nested aware? Is there any plan to
> add that to Metron in future?
> 
> Cheers,
> Ali
> 
> On Fri, Dec 22, 2017 at 12:46 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
> 
>> I believe right now you have to flatten.
>> The jsonMap parser does this.
>> 
>> 
>> On December 21, 2017 at 08:28:13, Ali Nazemian (alinazem...@gmail.com)
>> wrote:
>> 
>> Hi all,
>> 
>> 
>> We have recently faced some data sources that generate data in a nested
>> format. For example, AWS Cloudtrail generates data in the following JSON
>> format:
>> 
>> {
>> 
>> "Records": [
>> 
>> {
>> 
>> "eventVersion": *"2.0"*,
>> 
>> "userIdentity": {
>> 
>> "type": *"IAMUser"*,
>> 
>> "principalId": *"EX_PRINCIPAL_ID"*,
>> 
>> "arn": *"arn:aws:iam::123456789012:user/Alice"*,
>> 
>> "accessKeyId": *"EXAMPLE_KEY_ID"*,
>> 
>> "accountId": *"123456789012"*,
>> 
>> "userName": *"Alice"*
>> 
>> },
>> 
>> "eventTime": *"2014-03-07T21:22:54Z"*,
>> 
>> "eventSource": *"ec2.amazonaws.com <http://ec2.amazonaws.com>"*,
>> 
>> "eventName": *"StartInstances"*,
>> 
>> "awsRegion": *"us-east-2"*,
>> 
>> "sourceIPAddress": *"205.251.233.176"*,
>> 
>> "userAgent": *"ec2-api-tools 1.6.12.2"*,
>> 
>> "requestParameters": {
>> 
>> "instancesSet": {
>> 
>> "items": [
>> 
>> {
>> 
>> "instanceId": *"i-ebeaf9e2"*
>> 
>> }
>> 
>> ]
>> 
>> }
>> 
>> },
>> 
>> "responseElements": {
>> 
>> "instancesSet": {
>> 
>> "items": [
>> 
>> {
>> 
>> "instanceId": *"i-ebeaf9e2"*,
>> 
>> "currentState": {
>> 
>> "code": 0,
>> 
>> "name": *"pending"*
>> 
>> },
>> 
>> "previousState": {
>> 
>> "code": 80,
>> 
>> "name": *"stopped"*
>> 
>> }
>> 
>> }
>> 
>> ]
>> 
>> }
>> 
>> }
>> 
>> }
>> 
>> ]
>> 
>> }
>> 
>> 
>> We are able to make this as a flat JSON file. However, a nested object is
>> supported by data backends in Metron (ES, ORC, etc.), so I was wondering
>> whether with the current version of Metron we are able to index nested
>> documents or we have to make it flat?
>> 
>> 
>> 
>> Cheers,
>> 
>> Ali
>> 
>> 
> 
> 
> -- 
> A.Nazemian

Reply via email to