Thanks, Nick.
Can you give me more information on what the problem with Solr indexing was
at the first place? I've got some experience with Solr so I might be able
to help to fix that situation.
Regards,
Ali
On Mon, Apr 3, 2017 at 11:55 PM, Nick Allen <n...@nickallen.org> wrote:
> Up to this point, we have been making the assumption that we need to
> "flatten" complex data types like lists and maps before they get indexed.
> For example, a list like this...
>
> {
> users: [ mary, alice, bob ]
> }
>
>
> is flattened and ends up looking like this...
>
> {
> users.0: mary,
> users.1: alice,
> users.2: bob
> }
>
>
> The goal of the JIRA that I referenced is to make each indexer responsible
> for transforming the message in whatever way necessary to correctly index
> the data. This way enrichments and transformations that occur upstream
> don't have to worry about this.
>
> I *think* the specific issue is that Solr indexing may not work with
> complex data types like lists and maps in some scenarios. I *think*
> Elasticsearch indexing may be fine. Others may have more insight, but this
> is what I remember. It is probably worth the effort to validate this in
> your environment and see if any problems arise. It should be fairly simple
> to validate.
>
>
>
>
>
> On Sun, Apr 2, 2017 at 10:50 PM, Ali Nazemian <alinazem...@gmail.com>
> wrote:
>
>> Thank you very much, Nick. I was not aware of the fact that Metron does
>> not support the multi-value attribute. So, in this case, I need to have a
>> Stellar function to deal with splitting data and mapping to enrichment CF.
>> Is that correct?
>>
>> Regards,
>> Ali
>>
>> On Mon, Apr 3, 2017 at 6:31 AM, Nick Allen <n...@nickallen.org> wrote:
>>
>>> You could use the programmatic enrichment functions to do this. For
>>> instance, say you wanted to look-up the impacted users in a company
>>> 'phonebook' to get more information.
>>>
>>> 'impacted-user-0": ENRICHMENT_GET(''phonebook", GET(user_ids, 0), "tb",
>>> "cf")
>>>
>>> 'impacted-user-1": ENRICHMENT_GET(''phonebook", GET(user_ids, 1), "tb",
>>> "cf")
>>>
>>> "impacted-user-2": ENRICHMENT_GET(''phonebook", GET(user_ids, 2), "tb",
>>> "cf")
>>>
>>>
>>> Also note that there is an open JIRA to ensure that all of the index
>>> destinations can handle complex types in the message JSON. This may or may
>>> not impact your use case, but something to keep in mind.
>>>
>>> https://issues.apache.org/jira/browse/METRON-735
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <alinazem...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>>
>>>> I was wondering how I can achieve the following use case in the current
>>>> version of Metron?
>>>>
>>>>
>>>>
>>>> I want to have attributes in the Metron JSON object that are an array.
>>>> For example, if a threat is impacting multiple users, they are all
>>>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
>>>> to enrich the event with data that requires the user_id as a key in
>>>> enrichment stored in HBASE, how would I do this?
>>>>
>>>>
>>>> Cheers,
>>>> Ali
>>>>
>>>
>>>
>>
>>
>> --
>> A.Nazemian
>>
>
>
--
A.Nazemian
