I have no knowledge of specifics. I was not involved with the original problem. I just know that I had to flatten the output from threat triage based on these concerns as part of PR #438 [1].
[1] https://github.com/apache/incubator-metron/pull/438. On Mon, Apr 3, 2017 at 8:11 PM, Ali Nazemian <alinazem...@gmail.com> wrote: > Thanks, Nick. > > Can you give me more information on what the problem with Solr indexing > was at the first place? I've got some experience with Solr so I might be > able to help to fix that situation. > > Regards, > Ali > > On Mon, Apr 3, 2017 at 11:55 PM, Nick Allen <n...@nickallen.org> wrote: > >> Up to this point, we have been making the assumption that we need to >> "flatten" complex data types like lists and maps before they get indexed. >> For example, a list like this... >> >> { >> users: [ mary, alice, bob ] >> } >> >> >> is flattened and ends up looking like this... >> >> { >> users.0: mary, >> users.1: alice, >> users.2: bob >> } >> >> >> The goal of the JIRA that I referenced is to make each indexer >> responsible for transforming the message in whatever way necessary to >> correctly index the data. This way enrichments and transformations that >> occur upstream don't have to worry about this. >> >> I *think* the specific issue is that Solr indexing may not work with >> complex data types like lists and maps in some scenarios. I *think* >> Elasticsearch indexing may be fine. Others may have more insight, but this >> is what I remember. It is probably worth the effort to validate this in >> your environment and see if any problems arise. It should be fairly simple >> to validate. >> >> >> >> >> >> On Sun, Apr 2, 2017 at 10:50 PM, Ali Nazemian <alinazem...@gmail.com> >> wrote: >> >>> Thank you very much, Nick. I was not aware of the fact that Metron does >>> not support the multi-value attribute. So, in this case, I need to have a >>> Stellar function to deal with splitting data and mapping to enrichment CF. >>> Is that correct? >>> >>> Regards, >>> Ali >>> >>> On Mon, Apr 3, 2017 at 6:31 AM, Nick Allen <n...@nickallen.org> wrote: >>> >>>> You could use the programmatic enrichment functions to do this. For >>>> instance, say you wanted to look-up the impacted users in a company >>>> 'phonebook' to get more information. >>>> >>>> 'impacted-user-0": ENRICHMENT_GET(''phonebook", GET(user_ids, 0), "tb", >>>> "cf") >>>> >>>> 'impacted-user-1": ENRICHMENT_GET(''phonebook", GET(user_ids, 1), "tb", >>>> "cf") >>>> >>>> "impacted-user-2": ENRICHMENT_GET(''phonebook", GET(user_ids, 2), >>>> "tb", "cf") >>>> >>>> >>>> Also note that there is an open JIRA to ensure that all of the index >>>> destinations can handle complex types in the message JSON. This may or may >>>> not impact your use case, but something to keep in mind. >>>> >>>> https://issues.apache.org/jira/browse/METRON-735 >>>> >>>> >>>> >>>> >>>> >>>> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <alinazem...@gmail.com> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> >>>>> I was wondering how I can achieve the following use case in the >>>>> current version of Metron? >>>>> >>>>> >>>>> >>>>> I want to have attributes in the Metron JSON object that are an >>>>> array. For example, if a threat is impacting multiple users, they are all >>>>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want >>>>> to enrich the event with data that requires the user_id as a key in >>>>> enrichment stored in HBASE, how would I do this? >>>>> >>>>> >>>>> Cheers, >>>>> Ali >>>>> >>>> >>>> >>> >>> >>> -- >>> A.Nazemian >>> >> >> > > > -- > A.Nazemian >
