Thanks for doing this work, Casey. This is excellent
06.04.2017, 21:16, "Casey Stella" <ceste...@gmail.com>:
METRON-831, PR @ https://github.com/apache/incubator-metron/pull/517Just so we're clear, let's assume the following:
- Enrichment table called 'enrichments'
- Enrichment CF called 't'
- A message field called user_ids that is a list of user IDs
- The enrichment type for this HBase enrichment is 'et'
- The indicator in the HBase enrichment is a user ID
- You want a certain field out of the HBase enrichment data per user ID. Let's call that field 'login_time'
In order do that with METRON-831, you'd do the following to get the login time fields for the list of users. For the sake of simplicity, I'll break it into temporary variables:
- enriched_users := MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id, 'enrichments', 't') ) )
- login_times := MAP(enriched_users, &( enrichment : MAP_GET(enrichment, 'login_time') ) )
- MAP_GET here retrieves the value associated with the key 'login_time', which is the name.
Since you probably don't want intermediate values out there, you might want to smash that into one big statement (we need a way to remove temporary variables in stellar enrichments, btw):
- MAP(MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id, 'enrichments', 't') ) ), &( enrichment : MAP_GET(enrichment, 'login_time') ) )
On a side-note, it might be nice to have an optional arg to ENRICHMENT_GET that lets you specify just the fields to return. That would simplify the call to:
- MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id, 'enrichments', 't', ['login_time']) ) )
On Thu, Apr 6, 2017 at 8:10 PM, Casey Stella <ceste...@gmail.com> wrote:There'll be a JIRA and a PR tonight ;) It sprung from the keyboard. I've been waiting for a good reason for some time. hehOn Thu, Apr 6, 2017 at 8:08 PM, Otto Fowler <ottobackwa...@gmail.com> wrote:Is there a Jira for the MAP Casey?
On April 6, 2017 at 14:07:15, Casey Stella (ceste...@gmail.com) wrote:
Ok, so yeah, you've hit upon a limitation currently. Right now, via Stellar you can use ENRICHMENT_GET which takes the following parameters:
- enrichment_type - The enrichment type
- indicator - The string indicator to look up
- hbase_table - The HBase Table to use
- column_family - The Column Family to use
Right now we only accept a string for the indicator (which likely would be your user_id). You'd probably like to call ENRICHMENT_GET for each id in the user_id variable. We can't quite do that yet. There has been some talk about a MAP function created where you can apply a stellar function across a list of values. i.e. MAP( user_id, @ENRICHMENT_GET('et', $, 'enrichments', 't')) which would return a list containing the output of ENRICHMENT_GET for each call.There is another, more immediate change that could be made for this specific case. We could enable ENRICHMENT_GET to take a list of indicators as the second argument.Sorry, that doesn't exactly solve your problem in the immediate-case, but it provides some context for future fixes. ;) I don't suppose you know the length of the list beforehand, right? Even the maximum size?Casey
On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <alinazem...@gmail.com> wrote:
Hi all,
I was wondering how I can achieve the following use case in the current version of Metron?
I want to have attributes in the Metron JSON object that are an array. For example, if a threat is impacting multiple users, they are all contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want to enrich the event with data that requires the user_id as a key in enrichment stored in HBASE, how would I do this?
Cheers,Ali
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
