Correct me if I am wrong, Matt, but I believe that changing the project that you pass to the -pl switch will allow you to see exactly what Stellar functions would be available in each topology. You just have to refer to whichever project drives the topology.
This might help answer Ali's previous question as to what functions are available where. And of course, if some function is not available where it is needed, then it is a simply matter of changing the dependencies to make it available. For example, these functions are available from the Profiler. $ mvn exec:java \ -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \ -pl metron-analytics/metron-profiler ... Stellar, Go! Please note that functions are loading lazily in the background and will be unavailable until loaded fully. [Stellar]>>> Functions loaded, you may refer to functions now... %functions ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR And these functions are available in the Enrichment topology. $ mvn exec:java \ -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \ -pl metron-platform/metron-enrichment/ ... Stellar, Go! Please note that functions are loading lazily in the background and will be unavailable until loaded fully. [Stellar]>>> Functions loaded, you may refer to functions now... [Stellar]>>> %functions ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GEO_GET, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR For example, I can see from this that `GEO_GET` during Enrichment, but is not available in the Profiler right now. This makes sense because `GEO_GET` is defined in metron-enrichment. On Thu, May 25, 2017 at 1:58 PM, Nick Allen <[email protected]> wrote: > Thanks, Matt. That command-line doesn't work for me; even after updating > version number. Not sure why. This is what I tend to run after a build > and install of Metron. Maybe something is different in my environment. > > mvn clean install -DskipTests > > > mvn exec:java > -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" > -pl metron-platform/metron-common > > > For example... > > $ mvn exec:java > -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" > -pl metron-platform/metron-common > ... > > Stellar, Go! > Please note that functions are loading lazily in the background and will > be unavailable until loaded fully. > [Stellar]>>> Functions loaded, you may refer to functions now... > > [Stellar]>>> %functions > APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, > CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, > DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, > FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, IN_SUBNET, > IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, > LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, > MAP_GET, MONTH, PREPEND_IF_MISSING, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, > SPLIT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, > TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, > TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, > URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR > [Stellar]>>> > > > On Thu, May 25, 2017 at 1:31 PM, Matt Foley <[email protected]> > wrote: > >> Hi Ali, >> >> When writing Stellar statements, it is convenient to test them out in the >> REPL, which can be invoked via some variant of the commands in >> https://github.com/apache/metron/blob/master/metron-platform >> /metron-common/src/main/scripts/stellar , depending on the particular >> environment you’re working in . For instance, from the root of a git clone >> (that has already been successfully compiled), you can run: >> >> >> >> java -cp metron-platform/metron-enrichment/target/metron-enrichment- >> 0.3.1-uber.jar:metron-platform/metron-data-managemen >> t/target/metron-data-management-0.3.1.jar:metron-platform/ >> metron-common/target/metron-common-0.3.1.jar:metron- >> platform/metron-management/target/metron-management-0.3. >> 1.jar:metron-platform/metron-parsers/target/metron-parsers-0.3.1-uber.jar >> org.apache.metron.common.stellar.shell.StellarShell >> >> >> >> As you probably infer from the above, any jar that has @Stellar annotated >> classes has to be included in the cp, or those stellar operators won’t be >> available. The main jar is metron-common, which will give you the REPL and >> the “base” operator set, but many interesting ops are in those other jars. >> (If I missed one, just add it to the cp.) >> >> >> >> Once in the REPL you can type `%functions` to get a list of all available >> stellar operators. Basically, any class annotated with @Stellar will >> automatically be loaded, both in the REPL and in the installed runtime >> environment. >> >> >> >> Cheers, >> >> --Matt >> >> >> >> *From: *Otto Fowler <[email protected]> >> *Reply-To: *"[email protected]" <[email protected]> >> *Date: *Thursday, May 25, 2017 at 5:23 AM >> *To: *Nick Allen <[email protected]>, "[email protected]" < >> [email protected]> >> *Subject: *Re: Metron HBase conditional enrichment >> >> >> >> I think most of those restricted functions are in the metron-managment >> section. >> >> >> >> >> >> On May 25, 2017 at 07:27:24, Nick Allen ([email protected]) wrote: >> >> > everywhere I can use Stellar DSL, all of the functions have been >> implemented and ready to use? >> >> >> >> Generally, yes, you are right. >> >> >> >> I vaguely remember a couple instances of functions that are useful in the >> REPL only, but I cannot remember what those are right now. Hopefully we >> have those doc'd appropriately. >> >> >> >> >> >> >> >> On Wed, May 24, 2017 at 10:38 PM, Ali Nazemian <[email protected]> >> wrote: >> >> Hi Nick, >> >> >> >> I was not sure about the implementation, so does it generally mean >> everywhere I can use Stellar DSL, all of the functions have been >> implemented and ready to use? >> >> >> >> Cheers, >> >> Ali >> >> >> >> On Thu, May 25, 2017 at 2:52 AM, Nick Allen <[email protected]> wrote: >> >> > can I do the concatenation on the fly at the enrichment level, so I >> don't need to store this temp field in Elasticsearch/HDFS. >> >> >> >> Sure, absolutely. >> >> >> >> > Moreover, I need to have a conditional enrichment to say if you >> couldn't find any match for "tenant_name+device_type+device_name" lookup >> for "tenant_name+device_type+default_device". >> >> >> >> Yes, you can. You've got if/else, JOIN, IS_EMPTY, and others that should >> make implementing this logic pretty easy. >> >> >> >> >> >> >> >> >> >> On Tue, May 23, 2017 at 10:34 PM, Ali Nazemian <[email protected]> >> wrote: >> >> Hi, >> >> >> >> I was wondering how I can manage Stellar syntax to be aligned with the >> following structure for the HBase enrichment: >> >> >> >> HBase_row_key: tenant_name+device_type+device_name >> >> >> >> At the high-level, I need to create a separate field via a post-parse >> Stellar function to be a concatenation of tenan_name, device_type and >> device_name. Let's call this field "key". Basically, I need to do the >> enrichment on the "key" which would be corresponding to the HBase row key. >> My first question is *can I do the concatenation on the fly at the >> enrichment level, so I don't need to store this temp field in >> Elasticsearch/HDFS*. >> >> >> >> Moreover, I need to have a conditional enrichment to say if you couldn't >> find any match for "tenant_name+device_type+device_name" lookup for >> "tenant_name+device_type+default_device". The second question would be *how >> can I manage conditional enrichment like this one*. I would be really >> grateful if you can provide some example. >> >> >> >> Regards, >> >> Ali >> >> >> >> >> >> >> >> -- >> >> A.Nazemian >> >> >> >> >
