What happens when you... (1) Use SHELL_EDIT to paste a profiler definition (2) Use CONFIG_PUT it write that definition back to Zk (3) Use CONFIG_GET to then retrieve the profiler definition from Zk
Do you see the profiler definition that you created? If not, can you copy/paste your REPL session showing these steps? Thanks On Wed, Apr 5, 2017 at 1:29 PM, Laurens Vets <laur...@daemon.be> wrote: > Sorry about that, I'm referring to the following: > https://cwiki.apache.org/confluence/display/METRON/Profiling+Your+Streams > > On 2017-04-05 10:19, Nick Allen wrote: > > Sorry, but I am not sure what you are reading. Can you send a link? > > On Wed, Apr 5, 2017 at 1:16 PM, Laurens Vets <laur...@daemon.be> wrote: > >> Thank you for the help, but I still have the same problem. >> >> Followed the getting-started section step 2 to step 8 without any errors. >> Still no output from profilerConfig though. >> >> Also, from reading the Profiling Your Streams section under Metron >> Reference Application, I would think this would actually set up the >> profiler. Is there maybe a part missing there? >> >> On 2017-04-05 09:58, Nick Allen wrote: >> >> I agree with Casey. You just don't have a profile configured yet. >> >> These instructions should get you going. >> >> >> https://github.com/apache/incubator-metron/tree/master/metro >> n-analytics/metron-profiler#getting-started >> >> >> After following those instructions, your call to CONFIG_GET will retrieve >> the profiler definition. >> >> >> On Wed, Apr 5, 2017 at 12:52 PM, Casey Stella <ceste...@gmail.com> wrote: >> >>> the profiler.properties file is the properties file for the flux >>> topology. CONFIG_GET interacts with the config for the profiler in >>> zookeeper. Typically that's kept in >>> $METRON_HOME/config/zookeeper/profiler.json >>> and it's described in https://github.com/apache/incu >>> bator-metron/tree/master/metron-analytics/metron-profiler >>> >>> Are you sure you've pushed the profiler config to zookeeper? >>> >>> On Wed, Apr 5, 2017 at 12:21 PM, Laurens Vets <laur...@daemon.be> wrote: >>> >>>> I'm still having issues. I copied the metron-management-0.3.1.jar over >>>> to /usr/metron/0.3.1/lib and now "profilerConfig := CONFIG_GET("PROFILER")" >>>> works. However, when I type "profilerConfig", it's empty. >>>> >>>> The file "/usr/metron/0.3.1/config/profiler.properties" exists and has >>>> content. I would think that "profilerConfig" shows _something_? >>>> >>>> >>>> On 2017-04-04 15:46, Nick Allen wrote: >>>> >>>> The management functions, including CONFIG_GET, do not get deployed >>>> automatically right now. Copy the jar from the metron-management project >>>> to your VM at /usr/metron/0.3.1/lib then restart the REPL. >>>> >>>> On Apr 4, 2017 6:42 PM, "Laurens Vets" <laur...@daemon.be> wrote: >>>> >>>>> I'm doing the Metron reference application and I get stuck at >>>>> Profiling Your Streams. >>>>> >>>>> This works: >>>>> $ /usr/hdp/current/hbase-client/bin/hbase shell >>>>> hbase(main):001:0> create 'profiler', 'P' >>>>> >>>>> However, the next command fails: >>>>> >>>>> [root@node1 ~]# /usr/metron/0.3.1/bin/stellar -z node1:2181 >>>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". >>>>> SLF4J: Defaulting to no-operation (NOP) logger implementation >>>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for >>>>> further details. >>>>> Stellar, Go! >>>>> Please note that functions are loading lazily in the background and >>>>> will be unavailable until loaded fully. >>>>> {es.clustername=metron, es.ip=node1, es.port=9300, >>>>> es.date.format=yyyy.MM.dd.HH, parser.error.topic=indexing} >>>>> [Stellar]>>> Functions loaded, you may refer to functions now... >>>>> profilerConfig := CONFIG_GET("PROFILER") >>>>> [!] Unable to resolve function named 'CONFIG_GET'. Valid functions >>>>> are IS_DATE,MONTH,IS_DOMAIN,FORMAT,TRIM,WEEK_OF_MONTH,JOIN,MAP_G >>>>> ET,TO_INTEGER,OUTLIER_MAD_SCORE,YEAR,WEEK_OF_YEAR,PROTOCOL_T >>>>> O_NAME,ENDS_WITH,GET_FIRST,FILL_LEFT,STATS_MAX,DOMAIN_TO_TLD >>>>> ,PROFILE_FIXED,TO_STRING,DOMAIN_REMOVE_SUBDOMAINS,HLLP_MERGE >>>>> ,HLLP_CARDINALITY,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,STA >>>>> TS_SUM_SQUARES,DOMAIN_REMOVE_TLD,STATS_INIT,SYSTEM_ENV_GET,E >>>>> NRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,STATS_COUNT,GET,HLL >>>>> P_ADD,IS_INTEGER,DAY_OF_WEEK,STATS_BIN,IS_EMPTY,IS_EMAIL, >>>>> MAP_EXISTS,DAY_OF_YEAR,REGEXP_MATCH,PROFILE_WINDOW,TO_LOWER, >>>>> STATS_SKEWNESS,ENRICHMENT_EXISTS,URL_TO_PORT,DAY_OF_ >>>>> MONTH,LENGTH,GET_LAST,IN_SUBNET,FILL_RIGHT,SPLIT,MAAS_ >>>>> MODEL_APPLY,ABS,STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_ >>>>> GET,STATS_VARIANCE,STATS_ADD,TO_UPPER,OUTLIER_MAD_STATE_ >>>>> MERGE,TO_EPOCH_TIMESTAMP,MAAS_GET_ENDPOINT,BLOOM_EXISTS,BIN, >>>>> STATS_QUADRATIC_MEAN,BLOOM_INIT,BLOOM_ADD,URL_TO_PATH, >>>>> STATS_GEOMETRIC_MEAN,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,STATS_ >>>>> MIN,URL_TO_HOST,TO_FLOAT,STATS_SUM_LOGS,IS_URL,STATS_SUM,IS_IP,TO_LO >>>>> NG,STATS_MERGE,STATS_MEAN,STATS_PERCENTILE,STRING_ENTROPY,TO >>>>> _DOUBLE,PROFILE_GET >>>>> org.apache.metron.common.dsl.ParseException: Unable to resolve >>>>> function named 'CONFIG_GET'. Valid functions are >>>>> IS_DATE,MONTH,IS_DOMAIN,FORMAT,TRIM,WEEK_OF_MONTH,JOIN,MAP_G >>>>> ET,TO_INTEGER,OUTLIER_MAD_SCORE,YEAR,WEEK_OF_YEAR,PROTOCOL_T >>>>> O_NAME,ENDS_WITH,GET_FIRST,FILL_LEFT,STATS_MAX,DOMAIN_TO_TLD >>>>> ,PROFILE_FIXED,TO_STRING,DOMAIN_REMOVE_SUBDOMAINS,HLLP_MERGE >>>>> ,HLLP_CARDINALITY,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,STA >>>>> TS_SUM_SQUARES,DOMAIN_REMOVE_TLD,STATS_INIT,SYSTEM_ENV_GET,E >>>>> NRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,STATS_COUNT,GET,HLL >>>>> P_ADD,IS_INTEGER,DAY_OF_WEEK,STATS_BIN,IS_EMPTY,IS_EMAIL, >>>>> MAP_EXISTS,DAY_OF_YEAR,REGEXP_MATCH,PROFILE_WINDOW,TO_LOWER, >>>>> STATS_SKEWNESS,ENRICHMENT_EXISTS,URL_TO_PORT,DAY_OF_ >>>>> MONTH,LENGTH,GET_LAST,IN_SUBNET,FILL_RIGHT,SPLIT,MAAS_ >>>>> MODEL_APPLY,ABS,STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_ >>>>> GET,STATS_VARIANCE,STATS_ADD,TO_UPPER,OUTLIER_MAD_STATE_ >>>>> MERGE,TO_EPOCH_TIMESTAMP,MAAS_GET_ENDPOINT,BLOOM_EXISTS,BIN, >>>>> STATS_QUADRATIC_MEAN,BLOOM_INIT,BLOOM_ADD,URL_TO_PATH, >>>>> STATS_GEOMETRIC_MEAN,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,STATS_ >>>>> MIN,URL_TO_HOST,TO_FLOAT,STATS_SUM_LOGS,IS_URL,STATS_SUM,IS_IP,TO_LO >>>>> NG,STATS_MERGE,STATS_MEAN,STATS_PERCENTILE,STRING_ENTROPY,TO >>>>> _DOUBLE,PROFILE_GET >>>>> at org.apache.metron.common.stell >>>>> ar.StellarCompiler.resolveFunction(StellarCompiler.java:302) >>>>> at org.apache.metron.common.stell >>>>> ar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:261) >>>>> at org.apache.metron.common.stell >>>>> ar.generated.StellarParser$TransformationFuncContext.exitRul >>>>> e(StellarParser.java:1689) >>>>> at org.antlr.v4.runtime.Parser.tr >>>>> iggerExitRuleEvent(Parser.java:422) >>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) >>>>> at org.apache.metron.common.stell >>>>> ar.generated.StellarParser.transformation(StellarParser.java:160) >>>>> at org.apache.metron.common.stell >>>>> ar.BaseStellarProcessor.parse(BaseStellarProcessor.java:126) >>>>> at org.apache.metron.common.stell >>>>> ar.shell.StellarExecutor.execute(StellarExecutor.java:287) >>>>> at org.apache.metron.common.stell >>>>> ar.shell.StellarShell.executeStellar(StellarShell.java:361) >>>>> at org.apache.metron.common.stell >>>>> ar.shell.StellarShell.handleStellar(StellarShell.java:264) >>>>> at org.apache.metron.common.stell >>>>> ar.shell.StellarShell.execute(StellarShell.java:400) >>>>> at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53) >>>>> at java.util.concurrent.ThreadPoo >>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> at java.util.concurrent.ThreadPoo >>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Caused by: java.lang.IllegalStateException: Unknown function: >>>>> `CONFIG_GET` >>>>> at org.apache.metron.common.dsl.f >>>>> unctions.resolver.BaseFunctionResolver.apply(BaseFunctionRes >>>>> olver.java:106) >>>>> at org.apache.metron.common.dsl.f >>>>> unctions.resolver.BaseFunctionResolver.apply(BaseFunctionRes >>>>> olver.java:47) >>>>> at org.apache.metron.common.stell >>>>> ar.StellarCompiler.resolveFunction(StellarCompiler.java:297) >>>>> ... 14 more >>>>> [Stellar]>>> >>>>> >>>>> Any idea what might be going on here? Did the function change name or >>>>> something? >>>> >>>> >>>> >> >
