So Metron enrichment and indexer are not nested aware? Is there any plan to
add that to Metron in future?
Cheers,
Ali
On Fri, Dec 22, 2017 at 12:46 AM, Otto Fowler <ottobackwa...@gmail.com>
wrote:
> I believe right now you have to flatten.
> The jsonMap parser does this.
>
>
> On December 21, 2017 at 08:28:13, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> Hi all,
>
>
> We have recently faced some data sources that generate data in a nested
> format. For example, AWS Cloudtrail generates data in the following JSON
> format:
>
> {
>
> "Records": [
>
> {
>
> "eventVersion": *"2.0"*,
>
> "userIdentity": {
>
> "type": *"IAMUser"*,
>
> "principalId": *"EX_PRINCIPAL_ID"*,
>
> "arn": *"arn:aws:iam::123456789012:user/Alice"*,
>
> "accessKeyId": *"EXAMPLE_KEY_ID"*,
>
> "accountId": *"123456789012"*,
>
> "userName": *"Alice"*
>
> },
>
> "eventTime": *"2014-03-07T21:22:54Z"*,
>
> "eventSource": *"ec2.amazonaws.com <http://ec2.amazonaws.com>"*,
>
> "eventName": *"StartInstances"*,
>
> "awsRegion": *"us-east-2"*,
>
> "sourceIPAddress": *"205.251.233.176"*,
>
> "userAgent": *"ec2-api-tools 1.6.12.2"*,
>
> "requestParameters": {
>
> "instancesSet": {
>
> "items": [
>
> {
>
> "instanceId": *"i-ebeaf9e2"*
>
> }
>
> ]
>
> }
>
> },
>
> "responseElements": {
>
> "instancesSet": {
>
> "items": [
>
> {
>
> "instanceId": *"i-ebeaf9e2"*,
>
> "currentState": {
>
> "code": 0,
>
> "name": *"pending"*
>
> },
>
> "previousState": {
>
> "code": 80,
>
> "name": *"stopped"*
>
> }
>
> }
>
> ]
>
> }
>
> }
>
> }
>
> ]
>
> }
>
>
> We are able to make this as a flat JSON file. However, a nested object is
> supported by data backends in Metron (ES, ORC, etc.), so I was wondering
> whether with the current version of Metron we are able to index nested
> documents or we have to make it flat?
>
>
>
> Cheers,
>
> Ali
>
>
--
A.Nazemian
