What we need to have is a function to produce entity row_key based on
device format and tenant. We can put all the configurations in enrichment
config, but it makes it very complex because we will have different logic
for different customers.

I was thinking of storing that logic somewhere and create a function which
gets customer identifier, device name and a map of important event
properties. Although we need to modify logic, it won't be very frequently.
Putting that in the global config will not make any difference from the
operational complexity and it probably even makes it worse! However, an
extra latency per every event to load config would be a deal breaker for
us. Technically, it should be safe to use OBJECT_GET, but for the
configuration, it needs to parse it per event, right?



On Sat, Feb 3, 2018 at 1:49 AM, Otto Fowler <ottobackwa...@gmail.com> wrote:

> I think if we understand the use case, we may be able to think of a more
> general set of functionality for stellar to meet this and other cases.
>
> Will this configuration change?  Do you need to track that change without
> reloading?  How *much* is in the configuration?  Do we want
> people putting their own keys in the global and have to manage that
> upgrading ( I think it will get blown away )?
>
>
>
>
>
>
> On February 2, 2018 at 09:38:28, Casey Stella (ceste...@gmail.com) wrote:
>
> We use a guava cache to cache the data for 24 hours.  You can see how it's
> done here: https://github.com/apache/metron/blob/master/metron-
> platform/metron-enrichment/src/main/java/org/apache/
> metron/enrichment/stellar/ObjectGet.java
>
> We also do something like this in GEO_GET as well, but it's a bit more
> complex.
>
> On Fri, Feb 2, 2018 at 9:35 AM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>> I forgot we added OBJECT_GET. How does the caching work on that?
>>
>> Simn
>>
>>
>> On 2 Feb 2018, at 14:33, Nick Allen <n...@nickallen.org> wrote:
>>
>> There are many functions that use the global configuration.  For example,
>> GET_GEO in org.apache.metron.enrichment.stellar.GeoEnrichmentFunctions.
>> There might be a better example, but that is one is staring at me at the
>> moment.
>>
>> There is an OBJECT_GET function defined in 
>> org.apache.metron.enrichment.stellar.ObjectGet
>> that was purpose-built to retrieve files from HDFS.  If you wanted to
>> retrieve a configuration from HDFS that would be a good example (if you
>> can't just use that functions directly).
>>
>> On Fri, Feb 2, 2018 at 8:50 AM Ali Nazemian <alinazem...@gmail.com>
>> wrote:
>>
>>> Is there any Stellar function already been implemented in Metron that
>>> has a config file associated with it? I am trying to get an idea of how it
>>> works.
>>>
>>> On 3 Feb. 2018 00:44, "Simon Elliston Ball" <si...@simonellistonball.com>
>>> wrote:
>>>
>>>> Depends how you write the function class, but most likely, yes. Hence
>>>> global config option.
>>>>
>>>> Simon
>>>>
>>>> On 2 Feb 2018, at 13:42, Ali Nazemian <alinazem...@gmail.com> wrote:
>>>>
>>>> Does it mean every time the function gets called it will load the
>>>> config, but if I use the global one it will only read it one time and it
>>>> will be available in memory?
>>>>
>>>> On 2 Feb. 2018 21:53, "Simon Elliston Ball" <
>>>> si...@simonellistonball.com> wrote:
>>>>
>>>>> Shouldn’t be. The one this I would point out though is that you don’t
>>>>> necessarily know which supervisor you will be running from, so pulling 
>>>>> from
>>>>> HDFS would make sense. That said, the performance implications are 
>>>>> probably
>>>>> not great. A good option here would be to have the config available in the
>>>>> global config for example and refer to that, since most instances of
>>>>> stellar apply global config to their context.
>>>>>
>>>>> Simon
>>>>>
>>>>>
>>>>> On 2 Feb 2018, at 07:14, Ali Nazemian <alinazem...@gmail.com> wrote:
>>>>>
>>>>> Will be any problem if the Stellar function we want to implement need
>>>>> to load an external config file?
>>>>>
>>>>> Cheers,
>>>>> Ali
>>>>>
>>>>> On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian <alinazem...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks, All.
>>>>>>
>>>>>> Yes, Nick. It is highly related to our use case and the way that we
>>>>>> are going to enrich events with assets and vulnerability properties. It 
>>>>>> is
>>>>>> not a general case at all.
>>>>>>
>>>>>> Cheers,
>>>>>> Ali
>>>>>>
>>>>>> On Thu, Jan 18, 2018 at 5:43 AM, Matt Foley <ma...@apache.org> wrote:
>>>>>>
>>>>>>> Besides the example code Simon mentioned at
>>>>>>> https://github.com/apache/metron/tree/master/metron-stellar/
>>>>>>> stellar-3rd-party-example ,
>>>>>>> there is some documentation at http://metron.apache.org/curre
>>>>>>> nt-book/metron-stellar/stellar-common/3rdPartyStellar.html
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *From:* Nick Allen <n...@nickallen.org>
>>>>>>> *Reply-To:* "user@metron.apache.org" <user@metron.apache.org>
>>>>>>> *Date:* Wednesday, January 17, 2018 at 4:46 AM
>>>>>>> *To:* "user@metron.apache.org" <user@metron.apache.org>
>>>>>>> *Subject:* Re: Define a function that can be used in Stellar
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> If something we have already does not fit the bill, I would
>>>>>>> recommend creating that function in Java.   Since you described it as "a
>>>>>>> bit complex" and "the logic would be complicated" I don't see any value 
>>>>>>> in
>>>>>>> defining something like this in Stellar with named functions.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Best
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball <
>>>>>>> si...@simonellistonball.com> wrote:
>>>>>>>
>>>>>>> Have you looked at the recent TLSH functions in Stellar? We already
>>>>>>> have that for similarity preserving hashes.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Simon
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 17 Jan 2018, at 12:35, Ali Nazemian <alinazem...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> It is a bit complex. We want to create a function that accepts a
>>>>>>> list of arguments for an asset and generate an asset identifier that 
>>>>>>> can be
>>>>>>> used as a row_key for the enrichment store. The logic would be 
>>>>>>> complicated,
>>>>>>> though. We may need to include some sort of similarity aware hash 
>>>>>>> function
>>>>>>> as a part of this custom function.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 17, 2018 at 10:32 PM, Nick Allen <n...@nickallen.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Ali - Can you describe the logic that you are trying to perform?
>>>>>>> That would be useful as a use case to help drive a discussion around
>>>>>>> creating named functions in Stellar.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian <alinazem...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Thanks, Simon. We have already got a script to deal with classpath
>>>>>>> management for the parsers. We should be able to use it for this 
>>>>>>> extension
>>>>>>> as well.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Yeah, I agree. It will be much easier to define functions on the fly
>>>>>>> and use them afterwards. It could be defined as Lambda or custom 
>>>>>>> function.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Ali
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 17, 2018 at 9:42 PM, Simon Elliston Ball <
>>>>>>> si...@simonellistonball.com> wrote:
>>>>>>>
>>>>>>> https://github.com/apache/metron/tree/master/metron-stellar/
>>>>>>> stellar-3rd-party-example gives good details on how to add a
>>>>>>> stellar function.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Stellar will pick up an annotated function on its class path, so to
>>>>>>> add function there is no need to rebuild metron module, but you do need
>>>>>>> your modules on the classpath, and, pending 777, to deal with things 
>>>>>>> like
>>>>>>> class path clash in your dependencies.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Another idea worth discussion on the dev list is probably the notion
>>>>>>> of defining stellar functions in stellar, which would be a much simpler
>>>>>>> solution than custom java functions if you can already express you 
>>>>>>> logic in
>>>>>>> stellar.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Simon
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 17 Jan 2018, at 10:37, Ali Nazemian <alinazem...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi Simon,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Yes, that is exactly what we are looking for. Is there any example
>>>>>>> regarding adding a Stellar function in Java? Hopefully, we don't need to
>>>>>>> rebuild the corresponding modules for this?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Ali
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 17, 2018 at 8:40 PM, Simon Elliston Ball <
>>>>>>> si...@simonellistonball.com> wrote:
>>>>>>>
>>>>>>> At present you can certainly create custom stellar functions in
>>>>>>> Java. I’m guessing however that what you’re looking to do is create a 
>>>>>>> kind
>>>>>>> of function that combines a number of stellar functions to avoid
>>>>>>> repetition, or to ensure consistency of certain parameters for example. 
>>>>>>> Is
>>>>>>> that what you’re looking for? Maybe some sort of syntax to create a 
>>>>>>> named
>>>>>>> stellar function similar to the way we create lambdas?
>>>>>>>
>>>>>>> Simon
>>>>>>>
>>>>>>>
>>>>>>> > On 17 Jan 2018, at 07:25, Ali Nazemian <alinazem...@gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > Hi all,
>>>>>>> >
>>>>>>> > Is there any way that we can define a function that can be used
>>>>>>> rather than duplicating a logic multiple times?
>>>>>>> >
>>>>>>> > Cheers,
>>>>>>> > Ali
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> A.Nazemian
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> A.Nazemian
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> A.Nazemian
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> A.Nazemian
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> A.Nazemian
>>>>>
>>>>>
>>>>>
>>>>
>>
>


-- 
A.Nazemian

Reply via email to