Unfortunately,
this doc is
out-of-date
if you are running against the latest code.
Try the following. it
is equivalent to what you are trying to do.
PROFILE_GET("squid-miss", "192.168.66.121", PROFILE_FIXED(2, 'HOURS'))
Background:
An enhancement has been made to PROFILE_GET to allow much more flexible
retrieval of profile data. For example, if I want the profile data for the
past 4 Tuesdays excluding Holidays.
https://github.com/apache/incubator-metron/tree/master/metron-analytics/metron-profiler-client#profile-selectors
On Wed, Apr 5, 2017 at 5:02 PM, Laurens Vets <laur...@daemon.be> wrote:
> I'll see what I can do for the documents, maybe a bit of cleanup :)
>
> In the meantime, I've run into another issue.
>
> I've put in the 2 new profiles from https://cwiki.apache.org/
> confluence/display/METRON/Profiling+Your+Streams, squid-miss &
> url-length. I'm generating squid traffic and when I do "scan 'profiler'" in
> hbase shell, I see output for those.
>
> However, the next command fails:
>
> [Stellar]>>> profile := PROFILE_GET("squid-miss", "192.168.66.121", 2,
> "HOURS")
> [!] Unable to execute: Cannot cast java.lang.Integer to java.util.List
> org.apache.metron.common.dsl.ParseException: Unable to execute: Cannot
> cast java.lang.Integer to java.util.List
> at org.apache.metron.common.stellar.StellarCompiler.
> getResult(StellarCompiler.java:409)
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:127)
> at org.apache.metron.common.stellar.shell.StellarExecutor.
> execute(StellarExecutor.java:287)
> at org.apache.metron.common.stellar.shell.StellarShell.
> executeStellar(StellarShell.java:361)
> at org.apache.metron.common.stellar.shell.StellarShell.
> handleStellar(StellarShell.java:264)
> at org.apache.metron.common.stellar.shell.StellarShell.
> execute(StellarShell.java:400)
> at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassCastException: Cannot cast java.lang.Integer to
> java.util.List
> at java.lang.Class.cast(Class.java:3369)
> at org.apache.metron.common.utils.ConversionUtils.convert(
> ConversionUtils.java:42)
> at org.apache.metron.profiler.client.stellar.Util.getArg(
> Util.java:116)
> at org.apache.metron.profiler.client.stellar.GetProfile.
> apply(GetProfile.java:136)
> at org.apache.metron.common.stellar.StellarCompiler.
> exitTransformationFunc(StellarCompiler.java:267)
> at org.apache.metron.common.stellar.generated.StellarParser$
> TransformationFuncContext.exitRule(StellarParser.java:1689)
> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.
> java:422)
> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
> at org.apache.metron.common.stellar.generated.
> StellarParser.functions(StellarParser.java:1712)
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_operands(StellarParser.java:1846)
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr_mul(StellarParser.java:1609)
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr(StellarParser.java:1469)
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation_expr(StellarParser.java:308)
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation(StellarParser.java:149)
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:126)
> ... 8 more
> [Stellar]>>>
>
> Any idea what might be going on here?
>
> On 2017-04-05 12:21, Nick Allen wrote:
>
> No worries. If you are motivated, feel free to improve any of the docs
> that you find confusing. I know there is always room for improvement.
>
> On Wed, Apr 5, 2017 at 2:58 PM, Laurens Vets <laur...@daemon.be> wrote:
>
>> Yep, it works. Actually, now that I reread that whole section again I
>> think I've made the mistake of thinking that
>> /usr/metron/{$METRON_VERSION}/config/profiler.properties
>> was somehow related to getting output from "profilerConfig :=
>> CONFIG_GET("PROFILER")"...
>>
>> On 2017-04-05 11:54, Nick Allen wrote:
>>
>> Glad its working.
>>
>> The documentation does say that it will initially be empty. Which I
>> think matches your experience. No?
>>
>>
>> > To display the config (which should be initially empty just type:
>>
>>
>>
>>
>>
>> On Wed, Apr 5, 2017 at 2:50 PM, Laurens Vets <laur...@daemon.be> wrote:
>>
>>> This seems to work:
>>>
>>> (1) "profilerConfig := SHELL_EDIT()"
>>>
>>> Paste:
>>> {
>>> "profiles": [
>>> {
>>> "profile": "test",
>>> "foreach": "ip_src_addr",
>>> "init": { "count": "0" },
>>> "update": { "count": "count + 1" },
>>> "result": "count"
>>> }
>>> ]
>>> }
>>>
>>> (2) "CONFIG_PUT("PROFILER",profilerConfig)"
>>>
>>> (3) Quit Stellar prompt
>>> (4) Now when I do "CONFIG_GET("PROFILER")", I get output.
>>>
>>> So there's definitely something missing in the Metron Reference
>>> Application documentation. I'll continue and see how the rest goes. Thanks!
>>>
>>> On 2017-04-05 11:01, Nick Allen wrote:
>>>
>>> What happens when you...
>>>
>>> (1) Use SHELL_EDIT to paste a profiler definition
>>> (2) Use CONFIG_PUT it write that definition back to Zk
>>> (3) Use CONFIG_GET to then retrieve the profiler definition from Zk
>>>
>>> Do you see the profiler definition that you created? If not, can you
>>> copy/paste your REPL session showing these steps?
>>>
>>> Thanks
>>>
>>> On Wed, Apr 5, 2017 at 1:29 PM, Laurens Vets <laur...@daemon.be> wrote:
>>>
>>>> Sorry about that, I'm referring to the following:
>>>> https://cwiki.apache.org/confluence/display/METRON/Profiling
>>>> +Your+Streams
>>>>
>>>> On 2017-04-05 10:19, Nick Allen wrote:
>>>>
>>>> Sorry, but I am not sure what you are reading. Can you send a link?
>>>>
>>>> On Wed, Apr 5, 2017 at 1:16 PM, Laurens Vets <laur...@daemon.be> wrote:
>>>>
>>>>> Thank you for the help, but I still have the same problem.
>>>>>
>>>>> Followed the getting-started section step 2 to step 8 without any
>>>>> errors. Still no output from profilerConfig though.
>>>>>
>>>>> Also, from reading the Profiling Your Streams section under Metron
>>>>> Reference Application, I would think this would actually set up the
>>>>> profiler. Is there maybe a part missing there?
>>>>>
>>>>> On 2017-04-05 09:58, Nick Allen wrote:
>>>>>
>>>>> I agree with Casey. You just don't have a profile configured yet.
>>>>>
>>>>> These instructions should get you going.
>>>>>
>>>>>
>>>>> https://github.com/apache/incubator-metron/tree/master/metro
>>>>> n-analytics/metron-profiler#getting-started
>>>>>
>>>>>
>>>>> After following those instructions, your call to CONFIG_GET will
>>>>> retrieve the profiler definition.
>>>>>
>>>>>
>>>>> On Wed, Apr 5, 2017 at 12:52 PM, Casey Stella <ceste...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> the profiler.properties file is the properties file for the flux
>>>>>> topology. CONFIG_GET interacts with the config for the profiler in
>>>>>> zookeeper. Typically that's kept in
>>>>>> $METRON_HOME/config/zookeeper/profiler.json
>>>>>> and it's described in https://github.com/apache/incu
>>>>>> bator-metron/tree/master/metron-analytics/metron-profiler
>>>>>>
>>>>>> Are you sure you've pushed the profiler config to zookeeper?
>>>>>>
>>>>>> On Wed, Apr 5, 2017 at 12:21 PM, Laurens Vets <laur...@daemon.be>
>>>>>> wrote:
>>>>>>
>>>>>>> I'm still having issues. I copied the metron-management-0.3.1.jar
>>>>>>> over to /usr/metron/0.3.1/lib and now "profilerConfig :=
>>>>>>> CONFIG_GET("PROFILER")" works. However, when I type "profilerConfig",
>>>>>>> it's
>>>>>>> empty.
>>>>>>>
>>>>>>> The file "/usr/metron/0.3.1/config/profiler.properties" exists and
>>>>>>> has content. I would think that "profilerConfig" shows _something_?
>>>>>>>
>>>>>>>
>>>>>>> On 2017-04-04 15:46, Nick Allen wrote:
>>>>>>>
>>>>>>> The management functions, including CONFIG_GET, do not get deployed
>>>>>>> automatically right now. Copy the jar from the metron-management
>>>>>>> project
>>>>>>> to your VM at /usr/metron/0.3.1/lib then restart the REPL.
>>>>>>>
>>>>>>> On Apr 4, 2017 6:42 PM, "Laurens Vets" <laur...@daemon.be> wrote:
>>>>>>>
>>>>>>>> I'm doing the Metron reference application and I get stuck at
>>>>>>>> Profiling Your Streams.
>>>>>>>>
>>>>>>>> This works:
>>>>>>>> $ /usr/hdp/current/hbase-client/bin/hbase shell
>>>>>>>> hbase(main):001:0> create 'profiler', 'P'
>>>>>>>>
>>>>>>>> However, the next command fails:
>>>>>>>>
>>>>>>>> [root@node1 ~]# /usr/metron/0.3.1/bin/stellar -z node1:2181
>>>>>>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>>>>>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>>>>>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>>>>>>>> further details.
>>>>>>>> Stellar, Go!
>>>>>>>> Please note that functions are loading lazily in the background and
>>>>>>>> will be unavailable until loaded fully.
>>>>>>>> {es.clustername=metron, es.ip=node1, es.port=9300,
>>>>>>>> es.date.format=yyyy.MM.dd.HH, parser.error.topic=indexing}
>>>>>>>> [Stellar]>>> Functions loaded, you may refer to functions now...
>>>>>>>> profilerConfig := CONFIG_GET("PROFILER")
>>>>>>>> [!] Unable to resolve function named 'CONFIG_GET'. Valid functions
>>>>>>>> are IS_DATE,MONTH,IS_DOMAIN,FORMAT,TRIM,WEEK_OF_MONTH,JOIN,MAP_G
>>>>>>>> ET,TO_INTEGER,OUTLIER_MAD_SCORE,YEAR,WEEK_OF_YEAR,PROTOCOL_T
>>>>>>>> O_NAME,ENDS_WITH,GET_FIRST,FILL_LEFT,STATS_MAX,DOMAIN_TO_TLD
>>>>>>>> ,PROFILE_FIXED,TO_STRING,DOMAIN_REMOVE_SUBDOMAINS,HLLP_MERGE
>>>>>>>> ,HLLP_CARDINALITY,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,STA
>>>>>>>> TS_SUM_SQUARES,DOMAIN_REMOVE_TLD,STATS_INIT,SYSTEM_ENV_GET,E
>>>>>>>> NRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,STATS_COUNT,GET,HLL
>>>>>>>> P_ADD,IS_INTEGER,DAY_OF_WEEK,STATS_BIN,IS_EMPTY,IS_EMAIL,MAP
>>>>>>>> _EXISTS,DAY_OF_YEAR,REGEXP_MATCH,PROFILE_WINDOW,TO_LOWER,STA
>>>>>>>> TS_SKEWNESS,ENRICHMENT_EXISTS,URL_TO_PORT,DAY_OF_MONTH,LENGT
>>>>>>>> H,GET_LAST,IN_SUBNET,FILL_RIGHT,SPLIT,MAAS_MODEL_APPLY,ABS,
>>>>>>>> STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_GET,STATS_VARIANCE
>>>>>>>> ,STATS_ADD,TO_UPPER,OUTLIER_MAD_STATE_MERGE,TO_EPOCH_
>>>>>>>> TIMESTAMP,MAAS_GET_ENDPOINT,BLOOM_EXISTS,BIN,STATS_QUADRAT
>>>>>>>> IC_MEAN,BLOOM_INIT,BLOOM_ADD,URL_TO_PATH,STATS_GEOMETRIC_
>>>>>>>> MEAN,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,STATS_MIN,URL_TO_HOST
>>>>>>>> ,TO_FLOAT,STATS_SUM_LOGS,IS_URL,STATS_SUM,IS_IP,TO_LO
>>>>>>>> NG,STATS_MERGE,STATS_MEAN,STATS_PERCENTILE,STRING_ENTROPY,TO
>>>>>>>> _DOUBLE,PROFILE_GET
>>>>>>>> org.apache.metron.common.dsl.ParseException: Unable to resolve
>>>>>>>> function named 'CONFIG_GET'. Valid functions are
>>>>>>>> IS_DATE,MONTH,IS_DOMAIN,FORMAT,TRIM,WEEK_OF_MONTH,JOIN,MAP_G
>>>>>>>> ET,TO_INTEGER,OUTLIER_MAD_SCORE,YEAR,WEEK_OF_YEAR,PROTOCOL_T
>>>>>>>> O_NAME,ENDS_WITH,GET_FIRST,FILL_LEFT,STATS_MAX,DOMAIN_TO_TLD
>>>>>>>> ,PROFILE_FIXED,TO_STRING,DOMAIN_REMOVE_SUBDOMAINS,HLLP_MERGE
>>>>>>>> ,HLLP_CARDINALITY,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,STA
>>>>>>>> TS_SUM_SQUARES,DOMAIN_REMOVE_TLD,STATS_INIT,SYSTEM_ENV_GET,E
>>>>>>>> NRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,STATS_COUNT,GET,HLL
>>>>>>>> P_ADD,IS_INTEGER,DAY_OF_WEEK,STATS_BIN,IS_EMPTY,IS_EMAIL,MAP
>>>>>>>> _EXISTS,DAY_OF_YEAR,REGEXP_MATCH,PROFILE_WINDOW,TO_LOWER,STA
>>>>>>>> TS_SKEWNESS,ENRICHMENT_EXISTS,URL_TO_PORT,DAY_OF_MONTH,LENGT
>>>>>>>> H,GET_LAST,IN_SUBNET,FILL_RIGHT,SPLIT,MAAS_MODEL_APPLY,ABS,
>>>>>>>> STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_GET,STATS_VARIANCE
>>>>>>>> ,STATS_ADD,TO_UPPER,OUTLIER_MAD_STATE_MERGE,TO_EPOCH_
>>>>>>>> TIMESTAMP,MAAS_GET_ENDPOINT,BLOOM_EXISTS,BIN,STATS_QUADRAT
>>>>>>>> IC_MEAN,BLOOM_INIT,BLOOM_ADD,URL_TO_PATH,STATS_GEOMETRIC_
>>>>>>>> MEAN,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,STATS_MIN,URL_TO_HOST
>>>>>>>> ,TO_FLOAT,STATS_SUM_LOGS,IS_URL,STATS_SUM,IS_IP,TO_LO
>>>>>>>> NG,STATS_MERGE,STATS_MEAN,STATS_PERCENTILE,STRING_ENTROPY,TO
>>>>>>>> _DOUBLE,PROFILE_GET
>>>>>>>> at org.apache.metron.common.stell
>>>>>>>> ar.StellarCompiler.resolveFunction(StellarCompiler.java:302)
>>>>>>>> at org.apache.metron.common.stell
>>>>>>>> ar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:261)
>>>>>>>> at org.apache.metron.common.stell
>>>>>>>> ar.generated.StellarParser$TransformationFuncContext.exitRul
>>>>>>>> e(StellarParser.java:1689)
>>>>>>>> at org.antlr.v4.runtime.Parser.tr
>>>>>>>> iggerExitRuleEvent(Parser.java:422)
>>>>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>>>>> at org.apache.metron.common.stell
>>>>>>>> ar.generated.StellarParser.transformation(StellarParser.java:160)
>>>>>>>> at org.apache.metron.common.stell
>>>>>>>> ar.BaseStellarProcessor.parse(BaseStellarProcessor.java:126)
>>>>>>>> at org.apache.metron.common.stell
>>>>>>>> ar.shell.StellarExecutor.execute(StellarExecutor.java:287)
>>>>>>>> at org.apache.metron.common.stell
>>>>>>>> ar.shell.StellarShell.executeStellar(StellarShell.java:361)
>>>>>>>> at org.apache.metron.common.stell
>>>>>>>> ar.shell.StellarShell.handleStellar(StellarShell.java:264)
>>>>>>>> at org.apache.metron.common.stell
>>>>>>>> ar.shell.StellarShell.execute(StellarShell.java:400)
>>>>>>>> at org.jboss.aesh.console.AeshPro
>>>>>>>> cess.run(AeshProcess.java:53)
>>>>>>>> at java.util.concurrent.ThreadPoo
>>>>>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>>>> at java.util.concurrent.ThreadPoo
>>>>>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>> Caused by: java.lang.IllegalStateException: Unknown function:
>>>>>>>> `CONFIG_GET`
>>>>>>>> at org.apache.metron.common.dsl.f
>>>>>>>> unctions.resolver.BaseFunctionResolver.apply(BaseFunctionRes
>>>>>>>> olver.java:106)
>>>>>>>> at org.apache.metron.common.dsl.f
>>>>>>>> unctions.resolver.BaseFunctionResolver.apply(BaseFunctionRes
>>>>>>>> olver.java:47)
>>>>>>>> at org.apache.metron.common.stell
>>>>>>>> ar.StellarCompiler.resolveFunction(StellarCompiler.java:297)
>>>>>>>> ... 14 more
>>>>>>>> [Stellar]>>>
>>>>>>>>
>>>>>>>> Any idea what might be going on here? Did the function change name
>>>>>>>> or something?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>
