[jira] [Commented] (METRON-2330) Document Profiler "'global'" object
[
https://issues.apache.org/jira/browse/METRON-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012354#comment-17012354
]
Dima Kovalyov commented on METRON-2330:
---
Thank you for clarification Nick, I've updated documentation to reflect this.
> Document Profiler "'global'" object
> ---
>
> Key: METRON-2330
> URL: https://issues.apache.org/jira/browse/METRON-2330
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 1.7.1
>Reporter: Dima Kovalyov
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Dear Metron community,
>
> "[Statistics and Mathematical
> Functions|[https://metron.apache.org/current-book/metron-analytics/metron-statistics/index.html]];
> page makes use of:
> {code:java}
> "foreach": "'global'"
> {code}
> But nowhere on the internet, I was able to find any description of what it
> is, how it's working and how to troubleshoot it.
> The page mentions "We will capture a global statistical state for the
> {{value}} field and we will look back for a 5 minute window when computing
> the median." from which I can guess that 'global' represents the entire
> message instead of any particular field.
> Can you please shed some more light on it?
> In the comments, I'll post an example of my restle with it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (METRON-2330) Document Profiler "'global'" object
[
https://issues.apache.org/jira/browse/METRON-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986362#comment-16986362
]
Dima Kovalyov commented on METRON-2330:
---
Oh, I see, yeah that makes much more sense, thank you. I still wonder why the
specified profiler does not get committed either to HBase nor to ES.
When I define this one:
{code:java}
{
"profile": "distinct_auth_attempts_by_user",
"foreach": "user",
"onlyif": "source.type == 'auth' && auth_orientation != null &&
auth_orientation == 'LogOn' && user != null && LENGTH(user) > 0 && ip_dst_host
!= null && ip_dst_host != '?'",
"init": {
"total": "HLLP_INIT(5,6)"
},
"update": {
"total": "HLLP_ADD(total, ip_dst_host)"
},
"result": {
"profile": "total",
"triage": {
"total_count": "HLLP_CARDINALITY(total)"
}
}
}
{code}
It works just fine.
> Document Profiler "'global'" object
> ---
>
> Key: METRON-2330
> URL: https://issues.apache.org/jira/browse/METRON-2330
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 1.7.1
>Reporter: Dima Kovalyov
>Priority: Minor
>
> Dear Metron community,
>
> "[Statistics and Mathematical
> Functions|[https://metron.apache.org/current-book/metron-analytics/metron-statistics/index.html]];
> page makes use of:
> {code:java}
> "foreach": "'global'"
> {code}
> But nowhere on the internet, I was able to find any description of what it
> is, how it's working and how to troubleshoot it.
> The page mentions "We will capture a global statistical state for the
> {{value}} field and we will look back for a 5 minute window when computing
> the median." from which I can guess that 'global' represents the entire
> message instead of any particular field.
> Can you please shed some more light on it?
> In the comments, I'll post an example of my restle with it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (METRON-2330) Document Profiler "'global'" object
[
https://issues.apache.org/jira/browse/METRON-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986319#comment-16986319
]
Nick Allen commented on METRON-2330:
This example just uses a Stellar expression that returns the same value for all
messages applied to the profile. The example just uses a String named "global"
because the profile is building a single, global profile measurement/value. We
could have just as easily used an expression that returns the same value for
all messages, like 'cheese' or 'dima'.
Hope that helps
> Document Profiler "'global'" object
> ---
>
> Key: METRON-2330
> URL: https://issues.apache.org/jira/browse/METRON-2330
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 1.7.1
>Reporter: Dima Kovalyov
>Priority: Minor
>
> Dear Metron community,
>
> "[Statistics and Mathematical
> Functions|[https://metron.apache.org/current-book/metron-analytics/metron-statistics/index.html]];
> page makes use of:
> {code:java}
> "foreach": "'global'"
> {code}
> But nowhere on the internet, I was able to find any description of what it
> is, how it's working and how to troubleshoot it.
> The page mentions "We will capture a global statistical state for the
> {{value}} field and we will look back for a 5 minute window when computing
> the median." from which I can guess that 'global' represents the entire
> message instead of any particular field.
> Can you please shed some more light on it?
> In the comments, I'll post an example of my restle with it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (METRON-2330) Document Profiler "'global'" object
[
https://issues.apache.org/jira/browse/METRON-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16986262#comment-16986262
]
Dima Kovalyov commented on METRON-2330:
---
I have profiler running with:
{code:java}
{
"profiles": [
{
"profile": "distinct_auth_attempts_by_user_distribution",
"foreach": "'global'",
"onlyif": "profile == 'distinct_auth_attempts_by_user'",
"init": {
"s": "STATS_INIT()"
},
"update": {
"s": "STATS_ADD(s, total_count)"
},
"result": {
"profile": "total"
}
}
]
}
{code}
I stream following events directly to indexing topic:
{code:java}
{
"period.start": 157528920,
"period": 26254820,
"parallelenricher.splitter.end.ts": "1575290384733",
"total_count": 1,
"profile": "distinct_auth_attempts_by_user",
"is_alert": "true",
"parallelenricher.enrich.begin.ts": "1575290384733",
"source.type": "profiler",
"parallelenricher.splitter.begin.ts": "1575290384733",
"parallelenricher.enrich.end.ts": "1575290384733",
"period.end": 157528926,
"guid": "1346fe72-8b33-48c7-b377-714a234f0d78",
"entity": "C15131",
"timestamp": 1575290384721
}
{code}
After ingestion, these events are stored in ES profiler index. But they getting
ignored by the aforementioned profiler above.
When I execute the same profiler in REPL command line it is getting processed
just fine.
Please let me know if there is a better place to ask this question.
Thank you.
> Document Profiler "'global'" object
> ---
>
> Key: METRON-2330
> URL: https://issues.apache.org/jira/browse/METRON-2330
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 1.7.1
>Reporter: Dima Kovalyov
>Priority: Minor
>
> Dear Metron community,
>
> "[Statistics and Mathematical
> Functions|[https://metron.apache.org/current-book/metron-analytics/metron-statistics/index.html]];
> page makes use of:
> {code:java}
> "foreach": "'global'"
> {code}
> But nowhere on the internet, I was able to find any description of what it
> is, how it's working and how to troubleshoot it.
> The page mentions "We will capture a global statistical state for the
> {{value}} field and we will look back for a 5 minute window when computing
> the median." from which I can guess that 'global' represents the entire
> message instead of any particular field.
> Can you please shed some more light on it?
> In the comments, I'll post an example of my restle with it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
