There are many functions that use the global configuration. For example, GET_GEO in org.apache.metron.enrichment.stellar.GeoEnrichmentFunctions. There might be a better example, but that is one is staring at me at the moment.
There is an OBJECT_GET function defined in org.apache.metron.enrichment.stellar.ObjectGet that was purpose-built to retrieve files from HDFS. If you wanted to retrieve a configuration from HDFS that would be a good example (if you can't just use that functions directly). On Fri, Feb 2, 2018 at 8:50 AM Ali Nazemian <alinazem...@gmail.com> wrote: > Is there any Stellar function already been implemented in Metron that has > a config file associated with it? I am trying to get an idea of how it > works. > > On 3 Feb. 2018 00:44, "Simon Elliston Ball" <si...@simonellistonball.com> > wrote: > >> Depends how you write the function class, but most likely, yes. Hence >> global config option. >> >> Simon >> >> On 2 Feb 2018, at 13:42, Ali Nazemian <alinazem...@gmail.com> wrote: >> >> Does it mean every time the function gets called it will load the config, >> but if I use the global one it will only read it one time and it will be >> available in memory? >> >> On 2 Feb. 2018 21:53, "Simon Elliston Ball" <si...@simonellistonball.com> >> wrote: >> >>> Shouldn’t be. The one this I would point out though is that you don’t >>> necessarily know which supervisor you will be running from, so pulling from >>> HDFS would make sense. That said, the performance implications are probably >>> not great. A good option here would be to have the config available in the >>> global config for example and refer to that, since most instances of >>> stellar apply global config to their context. >>> >>> Simon >>> >>> >>> On 2 Feb 2018, at 07:14, Ali Nazemian <alinazem...@gmail.com> wrote: >>> >>> Will be any problem if the Stellar function we want to implement need to >>> load an external config file? >>> >>> Cheers, >>> Ali >>> >>> On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian <alinazem...@gmail.com> >>> wrote: >>> >>>> Thanks, All. >>>> >>>> Yes, Nick. It is highly related to our use case and the way that we are >>>> going to enrich events with assets and vulnerability properties. It is not >>>> a general case at all. >>>> >>>> Cheers, >>>> Ali >>>> >>>> On Thu, Jan 18, 2018 at 5:43 AM, Matt Foley <ma...@apache.org> wrote: >>>> >>>>> Besides the example code Simon mentioned at >>>>> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example >>>>> , >>>>> there is some documentation at >>>>> http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html >>>>> >>>>> >>>>> >>>>> *From: *Nick Allen <n...@nickallen.org> >>>>> *Reply-To: *"user@metron.apache.org" <user@metron.apache.org> >>>>> *Date: *Wednesday, January 17, 2018 at 4:46 AM >>>>> *To: *"user@metron.apache.org" <user@metron.apache.org> >>>>> *Subject: *Re: Define a function that can be used in Stellar >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> If something we have already does not fit the bill, I would recommend >>>>> creating that function in Java. Since you described it as "a bit >>>>> complex" >>>>> and "the logic would be complicated" I don't see any value in defining >>>>> something like this in Stellar with named functions. >>>>> >>>>> >>>>> >>>>> Best >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball < >>>>> si...@simonellistonball.com> wrote: >>>>> >>>>> Have you looked at the recent TLSH functions in Stellar? We already >>>>> have that for similarity preserving hashes. >>>>> >>>>> >>>>> >>>>> Simon >>>>> >>>>> >>>>> >>>>> >>>>> On 17 Jan 2018, at 12:35, Ali Nazemian <alinazem...@gmail.com> wrote: >>>>> >>>>> It is a bit complex. We want to create a function that accepts a list >>>>> of arguments for an asset and generate an asset identifier that can be >>>>> used >>>>> as a row_key for the enrichment store. The logic would be complicated, >>>>> though. We may need to include some sort of similarity aware hash function >>>>> as a part of this custom function. >>>>> >>>>> >>>>> >>>>> On Wed, Jan 17, 2018 at 10:32 PM, Nick Allen <n...@nickallen.org> >>>>> wrote: >>>>> >>>>> Ali - Can you describe the logic that you are trying to perform? That >>>>> would be useful as a use case to help drive a discussion around creating >>>>> named functions in Stellar. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian <alinazem...@gmail.com> >>>>> wrote: >>>>> >>>>> Thanks, Simon. We have already got a script to deal with classpath >>>>> management for the parsers. We should be able to use it for this extension >>>>> as well. >>>>> >>>>> >>>>> >>>>> Yeah, I agree. It will be much easier to define functions on the fly >>>>> and use them afterwards. It could be defined as Lambda or custom function. >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Ali >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Jan 17, 2018 at 9:42 PM, Simon Elliston Ball < >>>>> si...@simonellistonball.com> wrote: >>>>> >>>>> >>>>> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example >>>>> gives >>>>> good details on how to add a stellar function. >>>>> >>>>> >>>>> >>>>> Stellar will pick up an annotated function on its class path, so to >>>>> add function there is no need to rebuild metron module, but you do need >>>>> your modules on the classpath, and, pending 777, to deal with things like >>>>> class path clash in your dependencies. >>>>> >>>>> >>>>> >>>>> Another idea worth discussion on the dev list is probably the notion >>>>> of defining stellar functions in stellar, which would be a much simpler >>>>> solution than custom java functions if you can already express you logic >>>>> in >>>>> stellar. >>>>> >>>>> >>>>> >>>>> Simon >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 17 Jan 2018, at 10:37, Ali Nazemian <alinazem...@gmail.com> wrote: >>>>> >>>>> >>>>> >>>>> Hi Simon, >>>>> >>>>> >>>>> >>>>> Yes, that is exactly what we are looking for. Is there any example >>>>> regarding adding a Stellar function in Java? Hopefully, we don't need to >>>>> rebuild the corresponding modules for this? >>>>> >>>>> >>>>> >>>>> Cheers, >>>>> >>>>> Ali >>>>> >>>>> >>>>> >>>>> On Wed, Jan 17, 2018 at 8:40 PM, Simon Elliston Ball < >>>>> si...@simonellistonball.com> wrote: >>>>> >>>>> At present you can certainly create custom stellar functions in Java. >>>>> I’m guessing however that what you’re looking to do is create a kind of >>>>> function that combines a number of stellar functions to avoid repetition, >>>>> or to ensure consistency of certain parameters for example. Is that what >>>>> you’re looking for? Maybe some sort of syntax to create a named stellar >>>>> function similar to the way we create lambdas? >>>>> >>>>> Simon >>>>> >>>>> >>>>> > On 17 Jan 2018, at 07:25, Ali Nazemian <alinazem...@gmail.com> >>>>> wrote: >>>>> > >>>>> > Hi all, >>>>> > >>>>> > Is there any way that we can define a function that can be used >>>>> rather than duplicating a logic multiple times? >>>>> > >>>>> > Cheers, >>>>> > Ali >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> A.Nazemian >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> A.Nazemian >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> A.Nazemian >>>>> >>>>> >>>> >>>> >>>> -- >>>> A.Nazemian >>>> >>> >>> >>> >>> -- >>> A.Nazemian >>> >>> >>> >>
