No worries. If you are motivated, feel free to improve any of the docs
that you find confusing. I know there is always room for improvement.
On Wed, Apr 5, 2017 at 2:58 PM, Laurens Vets <laur...@daemon.be> wrote:
> Yep, it works. Actually, now that I reread that whole section again I
> think I've made the mistake of thinking that
> /usr/metron/{$METRON_VERSION}/config/profiler.properties
> was somehow related to getting output from "profilerConfig :=
> CONFIG_GET("PROFILER")"...
>
> On 2017-04-05 11:54, Nick Allen wrote:
>
> Glad its working.
>
> The documentation does say that it will initially be empty. Which I think
> matches your experience. No?
>
>
> > To display the config (which should be initially empty just type:
>
>
>
>
>
> On Wed, Apr 5, 2017 at 2:50 PM, Laurens Vets <laur...@daemon.be> wrote:
>
>> This seems to work:
>>
>> (1) "profilerConfig := SHELL_EDIT()"
>>
>> Paste:
>> {
>> "profiles": [
>> {
>> "profile": "test",
>> "foreach": "ip_src_addr",
>> "init": { "count": "0" },
>> "update": { "count": "count + 1" },
>> "result": "count"
>> }
>> ]
>> }
>>
>> (2) "CONFIG_PUT("PROFILER",profilerConfig)"
>>
>> (3) Quit Stellar prompt
>> (4) Now when I do "CONFIG_GET("PROFILER")", I get output.
>>
>> So there's definitely something missing in the Metron Reference
>> Application documentation. I'll continue and see how the rest goes. Thanks!
>>
>> On 2017-04-05 11:01, Nick Allen wrote:
>>
>> What happens when you...
>>
>> (1) Use SHELL_EDIT to paste a profiler definition
>> (2) Use CONFIG_PUT it write that definition back to Zk
>> (3) Use CONFIG_GET to then retrieve the profiler definition from Zk
>>
>> Do you see the profiler definition that you created? If not, can you
>> copy/paste your REPL session showing these steps?
>>
>> Thanks
>>
>> On Wed, Apr 5, 2017 at 1:29 PM, Laurens Vets <laur...@daemon.be> wrote:
>>
>>> Sorry about that, I'm referring to the following:
>>> https://cwiki.apache.org/confluence/display/METRON/Profiling
>>> +Your+Streams
>>>
>>> On 2017-04-05 10:19, Nick Allen wrote:
>>>
>>> Sorry, but I am not sure what you are reading. Can you send a link?
>>>
>>> On Wed, Apr 5, 2017 at 1:16 PM, Laurens Vets <laur...@daemon.be> wrote:
>>>
>>>> Thank you for the help, but I still have the same problem.
>>>>
>>>> Followed the getting-started section step 2 to step 8 without any
>>>> errors. Still no output from profilerConfig though.
>>>>
>>>> Also, from reading the Profiling Your Streams section under Metron
>>>> Reference Application, I would think this would actually set up the
>>>> profiler. Is there maybe a part missing there?
>>>>
>>>> On 2017-04-05 09:58, Nick Allen wrote:
>>>>
>>>> I agree with Casey. You just don't have a profile configured yet.
>>>>
>>>> These instructions should get you going.
>>>>
>>>>
>>>> https://github.com/apache/incubator-metron/tree/master/metro
>>>> n-analytics/metron-profiler#getting-started
>>>>
>>>>
>>>> After following those instructions, your call to CONFIG_GET will
>>>> retrieve the profiler definition.
>>>>
>>>>
>>>> On Wed, Apr 5, 2017 at 12:52 PM, Casey Stella <ceste...@gmail.com>
>>>> wrote:
>>>>
>>>>> the profiler.properties file is the properties file for the flux
>>>>> topology. CONFIG_GET interacts with the config for the profiler in
>>>>> zookeeper. Typically that's kept in
>>>>> $METRON_HOME/config/zookeeper/profiler.json
>>>>> and it's described in https://github.com/apache/incu
>>>>> bator-metron/tree/master/metron-analytics/metron-profiler
>>>>>
>>>>> Are you sure you've pushed the profiler config to zookeeper?
>>>>>
>>>>> On Wed, Apr 5, 2017 at 12:21 PM, Laurens Vets <laur...@daemon.be>
>>>>> wrote:
>>>>>
>>>>>> I'm still having issues. I copied the metron-management-0.3.1.jar
>>>>>> over to /usr/metron/0.3.1/lib and now "profilerConfig :=
>>>>>> CONFIG_GET("PROFILER")" works. However, when I type "profilerConfig",
>>>>>> it's
>>>>>> empty.
>>>>>>
>>>>>> The file "/usr/metron/0.3.1/config/profiler.properties" exists and
>>>>>> has content. I would think that "profilerConfig" shows _something_?
>>>>>>
>>>>>>
>>>>>> On 2017-04-04 15:46, Nick Allen wrote:
>>>>>>
>>>>>> The management functions, including CONFIG_GET, do not get deployed
>>>>>> automatically right now. Copy the jar from the metron-management project
>>>>>> to your VM at /usr/metron/0.3.1/lib then restart the REPL.
>>>>>>
>>>>>> On Apr 4, 2017 6:42 PM, "Laurens Vets" <laur...@daemon.be> wrote:
>>>>>>
>>>>>>> I'm doing the Metron reference application and I get stuck at
>>>>>>> Profiling Your Streams.
>>>>>>>
>>>>>>> This works:
>>>>>>> $ /usr/hdp/current/hbase-client/bin/hbase shell
>>>>>>> hbase(main):001:0> create 'profiler', 'P'
>>>>>>>
>>>>>>> However, the next command fails:
>>>>>>>
>>>>>>> [root@node1 ~]# /usr/metron/0.3.1/bin/stellar -z node1:2181
>>>>>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>>>>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>>>>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>>>>>>> further details.
>>>>>>> Stellar, Go!
>>>>>>> Please note that functions are loading lazily in the background and
>>>>>>> will be unavailable until loaded fully.
>>>>>>> {es.clustername=metron, es.ip=node1, es.port=9300,
>>>>>>> es.date.format=yyyy.MM.dd.HH, parser.error.topic=indexing}
>>>>>>> [Stellar]>>> Functions loaded, you may refer to functions now...
>>>>>>> profilerConfig := CONFIG_GET("PROFILER")
>>>>>>> [!] Unable to resolve function named 'CONFIG_GET'. Valid functions
>>>>>>> are IS_DATE,MONTH,IS_DOMAIN,FORMAT,TRIM,WEEK_OF_MONTH,JOIN,MAP_G
>>>>>>> ET,TO_INTEGER,OUTLIER_MAD_SCORE,YEAR,WEEK_OF_YEAR,PROTOCOL_T
>>>>>>> O_NAME,ENDS_WITH,GET_FIRST,FILL_LEFT,STATS_MAX,DOMAIN_TO_TLD
>>>>>>> ,PROFILE_FIXED,TO_STRING,DOMAIN_REMOVE_SUBDOMAINS,HLLP_MERGE
>>>>>>> ,HLLP_CARDINALITY,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,STA
>>>>>>> TS_SUM_SQUARES,DOMAIN_REMOVE_TLD,STATS_INIT,SYSTEM_ENV_GET,E
>>>>>>> NRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,STATS_COUNT,GET,HLL
>>>>>>> P_ADD,IS_INTEGER,DAY_OF_WEEK,STATS_BIN,IS_EMPTY,IS_EMAIL,MAP
>>>>>>> _EXISTS,DAY_OF_YEAR,REGEXP_MATCH,PROFILE_WINDOW,TO_LOWER,STA
>>>>>>> TS_SKEWNESS,ENRICHMENT_EXISTS,URL_TO_PORT,DAY_OF_MONTH,LENGT
>>>>>>> H,GET_LAST,IN_SUBNET,FILL_RIGHT,SPLIT,MAAS_MODEL_APPLY,
>>>>>>> ABS,STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_GET,STATS_
>>>>>>> VARIANCE,STATS_ADD,TO_UPPER,OUTLIER_MAD_STATE_MERGE,TO_
>>>>>>> EPOCH_TIMESTAMP,MAAS_GET_ENDPOINT,BLOOM_EXISTS,BIN,STATS_
>>>>>>> QUADRATIC_MEAN,BLOOM_INIT,BLOOM_ADD,URL_TO_PATH,STATS_GEOMET
>>>>>>> RIC_MEAN,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,STATS_MIN,URL_TO_
>>>>>>> HOST,TO_FLOAT,STATS_SUM_LOGS,IS_URL,STATS_SUM,IS_IP,TO_LO
>>>>>>> NG,STATS_MERGE,STATS_MEAN,STATS_PERCENTILE,STRING_ENTROPY,TO
>>>>>>> _DOUBLE,PROFILE_GET
>>>>>>> org.apache.metron.common.dsl.ParseException: Unable to resolve
>>>>>>> function named 'CONFIG_GET'. Valid functions are
>>>>>>> IS_DATE,MONTH,IS_DOMAIN,FORMAT,TRIM,WEEK_OF_MONTH,JOIN,MAP_G
>>>>>>> ET,TO_INTEGER,OUTLIER_MAD_SCORE,YEAR,WEEK_OF_YEAR,PROTOCOL_T
>>>>>>> O_NAME,ENDS_WITH,GET_FIRST,FILL_LEFT,STATS_MAX,DOMAIN_TO_TLD
>>>>>>> ,PROFILE_FIXED,TO_STRING,DOMAIN_REMOVE_SUBDOMAINS,HLLP_MERGE
>>>>>>> ,HLLP_CARDINALITY,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,STA
>>>>>>> TS_SUM_SQUARES,DOMAIN_REMOVE_TLD,STATS_INIT,SYSTEM_ENV_GET,E
>>>>>>> NRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,STATS_COUNT,GET,HLL
>>>>>>> P_ADD,IS_INTEGER,DAY_OF_WEEK,STATS_BIN,IS_EMPTY,IS_EMAIL,MAP
>>>>>>> _EXISTS,DAY_OF_YEAR,REGEXP_MATCH,PROFILE_WINDOW,TO_LOWER,STA
>>>>>>> TS_SKEWNESS,ENRICHMENT_EXISTS,URL_TO_PORT,DAY_OF_MONTH,LENGT
>>>>>>> H,GET_LAST,IN_SUBNET,FILL_RIGHT,SPLIT,MAAS_MODEL_APPLY,
>>>>>>> ABS,STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_GET,STATS_
>>>>>>> VARIANCE,STATS_ADD,TO_UPPER,OUTLIER_MAD_STATE_MERGE,TO_
>>>>>>> EPOCH_TIMESTAMP,MAAS_GET_ENDPOINT,BLOOM_EXISTS,BIN,STATS_
>>>>>>> QUADRATIC_MEAN,BLOOM_INIT,BLOOM_ADD,URL_TO_PATH,STATS_GEOMET
>>>>>>> RIC_MEAN,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,STATS_MIN,URL_TO_
>>>>>>> HOST,TO_FLOAT,STATS_SUM_LOGS,IS_URL,STATS_SUM,IS_IP,TO_LO
>>>>>>> NG,STATS_MERGE,STATS_MEAN,STATS_PERCENTILE,STRING_ENTROPY,TO
>>>>>>> _DOUBLE,PROFILE_GET
>>>>>>> at org.apache.metron.common.stell
>>>>>>> ar.StellarCompiler.resolveFunction(StellarCompiler.java:302)
>>>>>>> at org.apache.metron.common.stell
>>>>>>> ar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:261)
>>>>>>> at org.apache.metron.common.stell
>>>>>>> ar.generated.StellarParser$TransformationFuncContext.exitRul
>>>>>>> e(StellarParser.java:1689)
>>>>>>> at org.antlr.v4.runtime.Parser.tr
>>>>>>> iggerExitRuleEvent(Parser.java:422)
>>>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>>>> at org.apache.metron.common.stell
>>>>>>> ar.generated.StellarParser.transformation(StellarParser.java:160)
>>>>>>> at org.apache.metron.common.stell
>>>>>>> ar.BaseStellarProcessor.parse(BaseStellarProcessor.java:126)
>>>>>>> at org.apache.metron.common.stell
>>>>>>> ar.shell.StellarExecutor.execute(StellarExecutor.java:287)
>>>>>>> at org.apache.metron.common.stell
>>>>>>> ar.shell.StellarShell.executeStellar(StellarShell.java:361)
>>>>>>> at org.apache.metron.common.stell
>>>>>>> ar.shell.StellarShell.handleStellar(StellarShell.java:264)
>>>>>>> at org.apache.metron.common.stell
>>>>>>> ar.shell.StellarShell.execute(StellarShell.java:400)
>>>>>>> at org.jboss.aesh.console.AeshPro
>>>>>>> cess.run(AeshProcess.java:53)
>>>>>>> at java.util.concurrent.ThreadPoo
>>>>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>>> at java.util.concurrent.ThreadPoo
>>>>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> Caused by: java.lang.IllegalStateException: Unknown function:
>>>>>>> `CONFIG_GET`
>>>>>>> at org.apache.metron.common.dsl.f
>>>>>>> unctions.resolver.BaseFunctionResolver.apply(BaseFunctionRes
>>>>>>> olver.java:106)
>>>>>>> at org.apache.metron.common.dsl.f
>>>>>>> unctions.resolver.BaseFunctionResolver.apply(BaseFunctionRes
>>>>>>> olver.java:47)
>>>>>>> at org.apache.metron.common.stell
>>>>>>> ar.StellarCompiler.resolveFunction(StellarCompiler.java:297)
>>>>>>> ... 14 more
>>>>>>> [Stellar]>>>
>>>>>>>
>>>>>>> Any idea what might be going on here? Did the function change name
>>>>>>> or something?
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>
>
