[
https://issues.apache.org/jira/browse/METRON-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140510#comment-16140510
]
ASF GitHub Bot commented on METRON-1120:
----------------------------------------
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/708
> Profile's 'groupBy' Expression Has No Reference to Time
> -------------------------------------------------------
>
> Key: METRON-1120
> URL: https://issues.apache.org/jira/browse/METRON-1120
> Project: Metron
> Issue Type: Bug
> Reporter: Nick Allen
> Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> It is often the case that patterns and behaviors will differ based on
> calendar effects like day of week. For example, activity on a weekday can be
> very different from a weekend. The Profiler's "Group By" functionality is one
> way to account for calendar effects.
> This profile definition operates over any incoming telemetry that has an
> `ip_src_addr` and a `timestamp` field. It produces a profile that segments
> the data by day of week. It does by using a 'groupBy' expression to extract
> the day of week from the telemetry's `timestamp` field.
> {code}
> {
> "profiles": [
> {
> "profile": "calender-effects",
> "onlyif": "exists(ip_src_addr) and exists(timestamp)",
> "foreach": "ip_src_addr",
> "init": { "count": 0 },
> "update": { "count": "count + 1" },
> "result": "count",
> "groupBy": ["DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd
> HH:mm:ss', 'GMT'))"]
> }
> ]
> }
> {code}
> When retrieving profile data using the Profiler Client API, I only want to
> retrieve data from the same day of week to account for any calendar effects.
> The following example retrieves profile data only for Thursdays over the past
> 60 days.
> {code}
> >>> thursday := 5
> >>> PROFILE_GET("calendar-effects", "10.0.0.1", PROFILE_FIXED(60, "DAYS"),
> >>> [thursday])
> {code}
> h3. The Problem
> The `groupBy` expression only has access to the Profile's `result` value. It
> does not have any way to reference the current tick time in the Profiler.
> Here is an example showing the problem.
> Define the profile and a message.
> {code}
> [Stellar]>>> conf
> {
> "profiles": [
> {
> "profile": "calender-effects",
> "onlyif": "exists(ip_src_addr) and exists(timestamp)",
> "foreach": "ip_src_addr",
> "init": { "count": "0" },
> "update": { "count": "count + 1" },
> "result": "count",
> "groupBy": ["DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd
> HH:mm:ss', 'GMT'))"]
> }
> ]
> }
> [Stellar]>>> msg
> {
> "ip_src_addr": "10.0.0.1",
> "protocol": "HTTPS",
> "length": "10",
> "bytes_in": 234,
> "timestamp": "2017-08-17 09:00:00"
> }
> {code}
> Initialize the Profiler and apply the message a few times.
> {code}
> [Stellar]>>> p := PROFILER_INIT(conf)
> [Stellar]>>> PROFILER_APPLY(msg, p)
> org.apache.metron.profiler.StandAloneProfiler@9472c85
> [Stellar]>>> PROFILER_APPLY(msg, p)
> org.apache.metron.profiler.StandAloneProfiler@9472c85
> [Stellar]>>> PROFILER_APPLY(msg, p)
> org.apache.metron.profiler.StandAloneProfiler@9472c85
> {code}
> Flush the profile, which will trigger execution of the `groupBy` expression.
> {code}
> [Stellar]>>> PROFILER_FLUSH(p)
> [!] Bad 'groupBy' expression: Unexpected type: expected=Object, actual=null,
> expression=DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss',
> 'GMT')), profile=calender-effects, entity=10.0.0.1
> org.apache.metron.stellar.dsl.ParseException: Bad 'groupBy' expression:
> Unexpected type: expected=Object, actual=null,
> expression=DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss',
> 'GMT')), profile=calender-effects, entity=10.0.0.1
> at
> org.apache.metron.profiler.DefaultProfileBuilder.execute(DefaultProfileBuilder.java:257)
> at
> org.apache.metron.profiler.DefaultProfileBuilder.flush(DefaultProfileBuilder.java:159)
> at
> org.apache.metron.profiler.DefaultMessageDistributor.lambda$flush$0(DefaultMessageDistributor.java:101)
> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
> at
> org.apache.metron.profiler.DefaultMessageDistributor.flush(DefaultMessageDistributor.java:99)
> at
> org.apache.metron.profiler.StandAloneProfiler.flush(StandAloneProfiler.java:82)
> at
> org.apache.metron.profiler.client.stellar.ProfilerFunctions$ProfilerFlush.apply(ProfilerFunctions.java:191)
> at
> org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:556)
> at
> org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:160)
> at
> org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:152)
> at
> org.apache.metron.stellar.common.shell.StellarExecutor.execute(StellarExecutor.java:287)
> at
> org.apache.metron.stellar.common.shell.StellarShell.handleStellar(StellarShell.java:270)
> at
> org.apache.metron.stellar.common.shell.StellarShell.execute(StellarShell.java:409)
> at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException: Unexpected type:
> expected=Object, actual=null,
> expression=DAY_OF_WEEK(TO_EPOCH_TIMESTAMP(timestamp, 'yyyy-MM-dd HH:mm:ss',
> 'GMT'))
> at
> org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.execute(DefaultStellarStatefulExecutor.java:128)
> at
> org.apache.metron.profiler.DefaultProfileBuilder.lambda$execute$3(DefaultProfileBuilder.java:253)
> at java.util.ArrayList.forEach(ArrayList.java:1249)
> at
> org.apache.metron.profiler.DefaultProfileBuilder.execute(DefaultProfileBuilder.java:253)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
