METRON-831, PR @ https://github.com/apache/incubator-metron/pull/517
Just so we're clear, let's assume the following:
- Enrichment table called 'enrichments'
- Enrichment CF called 't'
- A message field called user_ids that is a list of user IDs
- The enrichment type for this HBase enrichment is 'et'
- The indicator in the HBase enrichment is a user ID
- You want a certain field out of the HBase enrichment data per user
ID. Let's call that field 'login_time'
In order do that with METRON-831, you'd do the following to get the login
time fields for the list of users. For the sake of simplicity, I'll break
it into temporary variables:
- enriched_users := MAP( user_ids, &( user_id : ENRICHMENT_GET('et',
user_id, 'enrichments', 't') ) )
- login_times := MAP(enriched_users, &( enrichment : MAP_GET(enrichment,
'login_time') ) )
- MAP_GET here retrieves the value associated with the key
'login_time', which is the name.
Since you probably don't want intermediate values out there, you might want
to smash that into one big statement (we need a way to remove temporary
variables in stellar enrichments, btw):
- MAP(MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id,
'enrichments', 't') ) ), &( enrichment : MAP_GET(enrichment, 'login_time')
) )
On a side-note, it might be nice to have an optional arg to ENRICHMENT_GET
that lets you specify just the fields to return. That would simplify the
call to:
- MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id,
'enrichments', 't', ['login_time']) ) )
On Thu, Apr 6, 2017 at 8:10 PM, Casey Stella <ceste...@gmail.com> wrote:
> There'll be a JIRA and a PR tonight ;) It sprung from the keyboard. I've
> been waiting for a good reason for some time. heh
>
> On Thu, Apr 6, 2017 at 8:08 PM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
>> Is there a Jira for the MAP Casey?
>>
>>
>> On April 6, 2017 at 14:07:15, Casey Stella (ceste...@gmail.com) wrote:
>>
>> Ok, so yeah, you've hit upon a limitation currently. Right now, via
>> Stellar you can use ENRICHMENT_GET which takes the following parameters:
>>
>> - enrichment_type - The enrichment type
>> - indicator - The string indicator to look up
>> - hbase_table - The HBase Table to use
>> - column_family - The Column Family to use
>>
>> Right now we only accept a string for the indicator (which likely would
>> be your user_id). You'd probably like to call ENRICHMENT_GET for each id
>> in the user_id variable. We can't quite do that yet. There has been some
>> talk about a MAP function created where you can apply a stellar function
>> across a list of values. i.e. MAP( user_id, @ENRICHMENT_GET('et', $,
>> 'enrichments', 't')) which would return a list containing the output of
>> ENRICHMENT_GET for each call.
>>
>> There is another, more immediate change that could be made for this
>> specific case. We could enable ENRICHMENT_GET to take a list of indicators
>> as the second argument.
>>
>> Sorry, that doesn't exactly solve your problem in the immediate-case, but
>> it provides some context for future fixes. ;) I don't suppose you know the
>> length of the list beforehand, right? Even the maximum size?
>>
>> Casey
>>
>>
>> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <alinazem...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>>
>>> I was wondering how I can achieve the following use case in the current
>>> version of Metron?
>>>
>>>
>>>
>>> I want to have attributes in the Metron JSON object that are an array.
>>> For example, if a threat is impacting multiple users, they are all
>>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
>>> to enrich the event with data that requires the user_id as a key in
>>> enrichment stored in HBASE, how would I do this?
>>>
>>>
>>> Cheers,
>>> Ali
>>>
>>
>>
>
