Glad its working.
The documentation does say that it will initially be empty. Which I think
matches your experience. No?
> To display the config (which should be initially empty just type:
On Wed, Apr 5, 2017 at 2:50 PM, Laurens Vets <laur...@daemon.be> wrote:
> This seems to work:
>
> (1) "profilerConfig := SHELL_EDIT()"
>
> Paste:
> {
> "profiles": [
> {
> "profile": "test",
> "foreach": "ip_src_addr",
> "init": { "count": "0" },
> "update": { "count": "count + 1" },
> "result": "count"
> }
> ]
> }
>
> (2) "CONFIG_PUT("PROFILER",profilerConfig)"
>
> (3) Quit Stellar prompt
> (4) Now when I do "CONFIG_GET("PROFILER")", I get output.
>
> So there's definitely something missing in the Metron Reference
> Application documentation. I'll continue and see how the rest goes. Thanks!
>
> On 2017-04-05 11:01, Nick Allen wrote:
>
> What happens when you...
>
> (1) Use SHELL_EDIT to paste a profiler definition
> (2) Use CONFIG_PUT it write that definition back to Zk
> (3) Use CONFIG_GET to then retrieve the profiler definition from Zk
>
> Do you see the profiler definition that you created? If not, can you
> copy/paste your REPL session showing these steps?
>
> Thanks
>
> On Wed, Apr 5, 2017 at 1:29 PM, Laurens Vets <laur...@daemon.be> wrote:
>
>> Sorry about that, I'm referring to the following:
>> https://cwiki.apache.org/confluence/display/METRON/Profiling+Your+Streams
>>
>> On 2017-04-05 10:19, Nick Allen wrote:
>>
>> Sorry, but I am not sure what you are reading. Can you send a link?
>>
>> On Wed, Apr 5, 2017 at 1:16 PM, Laurens Vets <laur...@daemon.be> wrote:
>>
>>> Thank you for the help, but I still have the same problem.
>>>
>>> Followed the getting-started section step 2 to step 8 without any
>>> errors. Still no output from profilerConfig though.
>>>
>>> Also, from reading the Profiling Your Streams section under Metron
>>> Reference Application, I would think this would actually set up the
>>> profiler. Is there maybe a part missing there?
>>>
>>> On 2017-04-05 09:58, Nick Allen wrote:
>>>
>>> I agree with Casey. You just don't have a profile configured yet.
>>>
>>> These instructions should get you going.
>>>
>>>
>>> https://github.com/apache/incubator-metron/tree/master/metro
>>> n-analytics/metron-profiler#getting-started
>>>
>>>
>>> After following those instructions, your call to CONFIG_GET will
>>> retrieve the profiler definition.
>>>
>>>
>>> On Wed, Apr 5, 2017 at 12:52 PM, Casey Stella <ceste...@gmail.com>
>>> wrote:
>>>
>>>> the profiler.properties file is the properties file for the flux
>>>> topology. CONFIG_GET interacts with the config for the profiler in
>>>> zookeeper. Typically that's kept in
>>>> $METRON_HOME/config/zookeeper/profiler.json
>>>> and it's described in https://github.com/apache/incu
>>>> bator-metron/tree/master/metron-analytics/metron-profiler
>>>>
>>>> Are you sure you've pushed the profiler config to zookeeper?
>>>>
>>>> On Wed, Apr 5, 2017 at 12:21 PM, Laurens Vets <laur...@daemon.be>
>>>> wrote:
>>>>
>>>>> I'm still having issues. I copied the metron-management-0.3.1.jar over
>>>>> to /usr/metron/0.3.1/lib and now "profilerConfig :=
>>>>> CONFIG_GET("PROFILER")"
>>>>> works. However, when I type "profilerConfig", it's empty.
>>>>>
>>>>> The file "/usr/metron/0.3.1/config/profiler.properties" exists and
>>>>> has content. I would think that "profilerConfig" shows _something_?
>>>>>
>>>>>
>>>>> On 2017-04-04 15:46, Nick Allen wrote:
>>>>>
>>>>> The management functions, including CONFIG_GET, do not get deployed
>>>>> automatically right now. Copy the jar from the metron-management project
>>>>> to your VM at /usr/metron/0.3.1/lib then restart the REPL.
>>>>>
>>>>> On Apr 4, 2017 6:42 PM, "Laurens Vets" <laur...@daemon.be> wrote:
>>>>>
>>>>>> I'm doing the Metron reference application and I get stuck at
>>>>>> Profiling Your Streams.
>>>>>>
>>>>>> This works:
>>>>>> $ /usr/hdp/current/hbase-client/bin/hbase shell
>>>>>> hbase(main):001:0> create 'profiler', 'P'
>>>>>>
>>>>>> However, the next command fails:
>>>>>>
>>>>>> [root@node1 ~]# /usr/metron/0.3.1/bin/stellar -z node1:2181
>>>>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>>>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>>>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>>>>>> further details.
>>>>>> Stellar, Go!
>>>>>> Please note that functions are loading lazily in the background and
>>>>>> will be unavailable until loaded fully.
>>>>>> {es.clustername=metron, es.ip=node1, es.port=9300,
>>>>>> es.date.format=yyyy.MM.dd.HH, parser.error.topic=indexing}
>>>>>> [Stellar]>>> Functions loaded, you may refer to functions now...
>>>>>> profilerConfig := CONFIG_GET("PROFILER")
>>>>>> [!] Unable to resolve function named 'CONFIG_GET'. Valid functions
>>>>>> are IS_DATE,MONTH,IS_DOMAIN,FORMAT,TRIM,WEEK_OF_MONTH,JOIN,MAP_G
>>>>>> ET,TO_INTEGER,OUTLIER_MAD_SCORE,YEAR,WEEK_OF_YEAR,PROTOCOL_T
>>>>>> O_NAME,ENDS_WITH,GET_FIRST,FILL_LEFT,STATS_MAX,DOMAIN_TO_TLD
>>>>>> ,PROFILE_FIXED,TO_STRING,DOMAIN_REMOVE_SUBDOMAINS,HLLP_MERGE
>>>>>> ,HLLP_CARDINALITY,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,STA
>>>>>> TS_SUM_SQUARES,DOMAIN_REMOVE_TLD,STATS_INIT,SYSTEM_ENV_GET,E
>>>>>> NRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,STATS_COUNT,GET,HLL
>>>>>> P_ADD,IS_INTEGER,DAY_OF_WEEK,STATS_BIN,IS_EMPTY,IS_EMAIL,MAP
>>>>>> _EXISTS,DAY_OF_YEAR,REGEXP_MATCH,PROFILE_WINDOW,TO_LOWER,STA
>>>>>> TS_SKEWNESS,ENRICHMENT_EXISTS,URL_TO_PORT,DAY_OF_MONTH,
>>>>>> LENGTH,GET_LAST,IN_SUBNET,FILL_RIGHT,SPLIT,MAAS_MODEL_
>>>>>> APPLY,ABS,STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_GET,
>>>>>> STATS_VARIANCE,STATS_ADD,TO_UPPER,OUTLIER_MAD_STATE_MERGE,
>>>>>> TO_EPOCH_TIMESTAMP,MAAS_GET_ENDPOINT,BLOOM_EXISTS,BIN,STAT
>>>>>> S_QUADRATIC_MEAN,BLOOM_INIT,BLOOM_ADD,URL_TO_PATH,STATS_
>>>>>> GEOMETRIC_MEAN,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,STATS_MIN,
>>>>>> URL_TO_HOST,TO_FLOAT,STATS_SUM_LOGS,IS_URL,STATS_SUM,IS_IP,TO_LO
>>>>>> NG,STATS_MERGE,STATS_MEAN,STATS_PERCENTILE,STRING_ENTROPY,TO
>>>>>> _DOUBLE,PROFILE_GET
>>>>>> org.apache.metron.common.dsl.ParseException: Unable to resolve
>>>>>> function named 'CONFIG_GET'. Valid functions are
>>>>>> IS_DATE,MONTH,IS_DOMAIN,FORMAT,TRIM,WEEK_OF_MONTH,JOIN,MAP_G
>>>>>> ET,TO_INTEGER,OUTLIER_MAD_SCORE,YEAR,WEEK_OF_YEAR,PROTOCOL_T
>>>>>> O_NAME,ENDS_WITH,GET_FIRST,FILL_LEFT,STATS_MAX,DOMAIN_TO_TLD
>>>>>> ,PROFILE_FIXED,TO_STRING,DOMAIN_REMOVE_SUBDOMAINS,HLLP_MERGE
>>>>>> ,HLLP_CARDINALITY,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,STA
>>>>>> TS_SUM_SQUARES,DOMAIN_REMOVE_TLD,STATS_INIT,SYSTEM_ENV_GET,E
>>>>>> NRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,STATS_COUNT,GET,HLL
>>>>>> P_ADD,IS_INTEGER,DAY_OF_WEEK,STATS_BIN,IS_EMPTY,IS_EMAIL,MAP
>>>>>> _EXISTS,DAY_OF_YEAR,REGEXP_MATCH,PROFILE_WINDOW,TO_LOWER,STA
>>>>>> TS_SKEWNESS,ENRICHMENT_EXISTS,URL_TO_PORT,DAY_OF_MONTH,
>>>>>> LENGTH,GET_LAST,IN_SUBNET,FILL_RIGHT,SPLIT,MAAS_MODEL_
>>>>>> APPLY,ABS,STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_GET,
>>>>>> STATS_VARIANCE,STATS_ADD,TO_UPPER,OUTLIER_MAD_STATE_MERGE,
>>>>>> TO_EPOCH_TIMESTAMP,MAAS_GET_ENDPOINT,BLOOM_EXISTS,BIN,STAT
>>>>>> S_QUADRATIC_MEAN,BLOOM_INIT,BLOOM_ADD,URL_TO_PATH,STATS_
>>>>>> GEOMETRIC_MEAN,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,STATS_MIN,
>>>>>> URL_TO_HOST,TO_FLOAT,STATS_SUM_LOGS,IS_URL,STATS_SUM,IS_IP,TO_LO
>>>>>> NG,STATS_MERGE,STATS_MEAN,STATS_PERCENTILE,STRING_ENTROPY,TO
>>>>>> _DOUBLE,PROFILE_GET
>>>>>> at org.apache.metron.common.stell
>>>>>> ar.StellarCompiler.resolveFunction(StellarCompiler.java:302)
>>>>>> at org.apache.metron.common.stell
>>>>>> ar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:261)
>>>>>> at org.apache.metron.common.stell
>>>>>> ar.generated.StellarParser$TransformationFuncContext.exitRul
>>>>>> e(StellarParser.java:1689)
>>>>>> at org.antlr.v4.runtime.Parser.tr
>>>>>> iggerExitRuleEvent(Parser.java:422)
>>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>>> at org.apache.metron.common.stell
>>>>>> ar.generated.StellarParser.transformation(StellarParser.java:160)
>>>>>> at org.apache.metron.common.stell
>>>>>> ar.BaseStellarProcessor.parse(BaseStellarProcessor.java:126)
>>>>>> at org.apache.metron.common.stell
>>>>>> ar.shell.StellarExecutor.execute(StellarExecutor.java:287)
>>>>>> at org.apache.metron.common.stell
>>>>>> ar.shell.StellarShell.executeStellar(StellarShell.java:361)
>>>>>> at org.apache.metron.common.stell
>>>>>> ar.shell.StellarShell.handleStellar(StellarShell.java:264)
>>>>>> at org.apache.metron.common.stell
>>>>>> ar.shell.StellarShell.execute(StellarShell.java:400)
>>>>>> at org.jboss.aesh.console.AeshPro
>>>>>> cess.run(AeshProcess.java:53)
>>>>>> at java.util.concurrent.ThreadPoo
>>>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>> at java.util.concurrent.ThreadPoo
>>>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> Caused by: java.lang.IllegalStateException: Unknown function:
>>>>>> `CONFIG_GET`
>>>>>> at org.apache.metron.common.dsl.f
>>>>>> unctions.resolver.BaseFunctionResolver.apply(BaseFunctionRes
>>>>>> olver.java:106)
>>>>>> at org.apache.metron.common.dsl.f
>>>>>> unctions.resolver.BaseFunctionResolver.apply(BaseFunctionRes
>>>>>> olver.java:47)
>>>>>> at org.apache.metron.common.stell
>>>>>> ar.StellarCompiler.resolveFunction(StellarCompiler.java:297)
>>>>>> ... 14 more
>>>>>> [Stellar]>>>
>>>>>>
>>>>>> Any idea what might be going on here? Did the function change name or
>>>>>> something?
>>>>>
>>>>>
>>>>>
>>>
>>
>
