I forgot we added OBJECT_GET. How does the caching work on that? Simn
> On 2 Feb 2018, at 14:33, Nick Allen <n...@nickallen.org> wrote: > > There are many functions that use the global configuration. For example, > GET_GEO in org.apache.metron.enrichment.stellar.GeoEnrichmentFunctions. > There might be a better example, but that is one is staring at me at the > moment. > > There is an OBJECT_GET function defined in > org.apache.metron.enrichment.stellar.ObjectGet that was purpose-built to > retrieve files from HDFS. If you wanted to retrieve a configuration from > HDFS that would be a good example (if you can't just use that functions > directly). > > On Fri, Feb 2, 2018 at 8:50 AM Ali Nazemian <alinazem...@gmail.com > <mailto:alinazem...@gmail.com>> wrote: > Is there any Stellar function already been implemented in Metron that has a > config file associated with it? I am trying to get an idea of how it works. > > On 3 Feb. 2018 00:44, "Simon Elliston Ball" <si...@simonellistonball.com > <mailto:si...@simonellistonball.com>> wrote: > Depends how you write the function class, but most likely, yes. Hence global > config option. > > Simon > >> On 2 Feb 2018, at 13:42, Ali Nazemian <alinazem...@gmail.com >> <mailto:alinazem...@gmail.com>> wrote: >> >> Does it mean every time the function gets called it will load the config, >> but if I use the global one it will only read it one time and it will be >> available in memory? >> >> On 2 Feb. 2018 21:53, "Simon Elliston Ball" <si...@simonellistonball.com >> <mailto:si...@simonellistonball.com>> wrote: >> Shouldn’t be. The one this I would point out though is that you don’t >> necessarily know which supervisor you will be running from, so pulling from >> HDFS would make sense. That said, the performance implications are probably >> not great. A good option here would be to have the config available in the >> global config for example and refer to that, since most instances of stellar >> apply global config to their context. >> >> Simon >> >> >>> On 2 Feb 2018, at 07:14, Ali Nazemian <alinazem...@gmail.com >>> <mailto:alinazem...@gmail.com>> wrote: >>> >>> Will be any problem if the Stellar function we want to implement need to >>> load an external config file? >>> >>> Cheers, >>> Ali >>> >>> On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian <alinazem...@gmail.com >>> <mailto:alinazem...@gmail.com>> wrote: >>> Thanks, All. >>> >>> Yes, Nick. It is highly related to our use case and the way that we are >>> going to enrich events with assets and vulnerability properties. It is not >>> a general case at all. >>> >>> Cheers, >>> Ali >>> >>> On Thu, Jan 18, 2018 at 5:43 AM, Matt Foley <ma...@apache.org >>> <mailto:ma...@apache.org>> wrote: >>> Besides the example code Simon mentioned at >>> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example >>> >>> <https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example> >>> , >>> there is some documentation at >>> http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html >>> >>> <http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html> >>> >>> >>> From: Nick Allen <n...@nickallen.org <mailto:n...@nickallen.org>> >>> Reply-To: "user@metron.apache.org <mailto:user@metron.apache.org>" >>> <user@metron.apache.org <mailto:user@metron.apache.org>> >>> Date: Wednesday, January 17, 2018 at 4:46 AM >>> To: "user@metron.apache.org <mailto:user@metron.apache.org>" >>> <user@metron.apache.org <mailto:user@metron.apache.org>> >>> Subject: Re: Define a function that can be used in Stellar >>> >>> >>> >>> >>> >>> >>> >>> If something we have already does not fit the bill, I would recommend >>> creating that function in Java. Since you described it as "a bit complex" >>> and "the logic would be complicated" I don't see any value in defining >>> something like this in Stellar with named functions. >>> >>> >>> >>> Best >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball >>> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote: >>> >>> Have you looked at the recent TLSH functions in Stellar? We already have >>> that for similarity preserving hashes. >>> >>> >>> >>> Simon >>> >>> >>> >>> >>> On 17 Jan 2018, at 12:35, Ali Nazemian <alinazem...@gmail.com >>> <mailto:alinazem...@gmail.com>> wrote: >>> >>> It is a bit complex. We want to create a function that accepts a list of >>> arguments for an asset and generate an asset identifier that can be used as >>> a row_key for the enrichment store. The logic would be complicated, though. >>> We may need to include some sort of similarity aware hash function as a >>> part of this custom function. >>> >>> >>> >>> On Wed, Jan 17, 2018 at 10:32 PM, Nick Allen <n...@nickallen.org >>> <mailto:n...@nickallen.org>> wrote: >>> >>> Ali - Can you describe the logic that you are trying to perform? That would >>> be useful as a use case to help drive a discussion around creating named >>> functions in Stellar. >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian <alinazem...@gmail.com >>> <mailto:alinazem...@gmail.com>> wrote: >>> >>> Thanks, Simon. We have already got a script to deal with classpath >>> management for the parsers. We should be able to use it for this extension >>> as well. >>> >>> >>> >>> Yeah, I agree. It will be much easier to define functions on the fly and >>> use them afterwards. It could be defined as Lambda or custom function. >>> >>> >>> >>> Regards, >>> >>> Ali >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Jan 17, 2018 at 9:42 PM, Simon Elliston Ball >>> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote: >>> >>> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example >>> >>> <https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example> >>> gives good details on how to add a stellar function. >>> >>> >>> >>> Stellar will pick up an annotated function on its class path, so to add >>> function there is no need to rebuild metron module, but you do need your >>> modules on the classpath, and, pending 777, to deal with things like class >>> path clash in your dependencies. >>> >>> >>> >>> Another idea worth discussion on the dev list is probably the notion of >>> defining stellar functions in stellar, which would be a much simpler >>> solution than custom java functions if you can already express you logic in >>> stellar. >>> >>> >>> >>> Simon >>> >>> >>> >>> >>> >>> >>> On 17 Jan 2018, at 10:37, Ali Nazemian <alinazem...@gmail.com >>> <mailto:alinazem...@gmail.com>> wrote: >>> >>> >>> >>> Hi Simon, >>> >>> >>> >>> Yes, that is exactly what we are looking for. Is there any example >>> regarding adding a Stellar function in Java? Hopefully, we don't need to >>> rebuild the corresponding modules for this? >>> >>> >>> >>> Cheers, >>> >>> Ali >>> >>> >>> >>> On Wed, Jan 17, 2018 at 8:40 PM, Simon Elliston Ball >>> <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote: >>> >>> At present you can certainly create custom stellar functions in Java. I’m >>> guessing however that what you’re looking to do is create a kind of >>> function that combines a number of stellar functions to avoid repetition, >>> or to ensure consistency of certain parameters for example. Is that what >>> you’re looking for? Maybe some sort of syntax to create a named stellar >>> function similar to the way we create lambdas? >>> >>> Simon >>> >>> >>> > On 17 Jan 2018, at 07:25, Ali Nazemian <alinazem...@gmail.com >>> > <mailto:alinazem...@gmail.com>> wrote: >>> > >>> > Hi all, >>> > >>> > Is there any way that we can define a function that can be used rather >>> > than duplicating a logic multiple times? >>> > >>> > Cheers, >>> > Ali >>> >>> >>> >>> >>> >>> >>> -- >>> >>> A.Nazemian >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> A.Nazemian >>> >>> >>> >>> >>> >>> >>> -- >>> >>> A.Nazemian >>> >>> >>> >>> >>> -- >>> A.Nazemian >>> >>> >>> >>> -- >>> A.Nazemian >> >
