[
https://issues.apache.org/jira/browse/METRON-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838564#comment-15838564
]
ASF GitHub Bot commented on METRON-668:
---------------------------------------
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/420
METRON-668: Remove the "tickUpdate" profile config and make the "init"
phase not reset variables
Please see description at
[METRON-668](https://issues.apache.org/jira/browse/METRON-668) for a full
description.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron METRON-668
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/420.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #420
----
commit 7cb5c60462448b6c35a9d1def58489903a649834
Author: cstella <ceste...@gmail.com>
Date: 2017-01-19T21:58:27Z
METRON-668: Remove the 'tickUpdate' profile config and make the 'init'
phase not reset variables
commit ad94ed4916605b1416c62463fc2de96f76c85f9f
Author: cstella <ceste...@gmail.com>
Date: 2017-01-19T22:00:47Z
Fixed docs
commit a99c013a3e850a26720ad9a7bfede226660e18ed
Author: cstella <ceste...@gmail.com>
Date: 2017-01-23T19:07:24Z
Updating unit tests to function properly.
commit 78a472fda1e65c0679089127d149582f861c82a7
Author: cstella <ceste...@gmail.com>
Date: 2017-01-23T20:56:20Z
TEMPORARY UPDATE TO SEE SOMETHING, DO NOT MERGE YET
commit 2d4aa56324f86cc5e3b73d486a2fde008ec211b8
Author: cstella <ceste...@gmail.com>
Date: 2017-01-23T21:24:32Z
updating one more time to indicate writes vs flush
commit b78e31fef6244aba98b369553125c8c667ff0a35
Author: cstella <ceste...@gmail.com>
Date: 2017-01-23T21:27:02Z
updating again.
commit 1cd6fa49b7d950e7433733c0ff683d42079dbca7
Author: cstella <ceste...@gmail.com>
Date: 2017-01-25T20:01:35Z
better logging.
commit 80a47ba0c402e70f0020447aff69e5feddef1dce
Author: cstella <ceste...@gmail.com>
Date: 2017-01-25T20:39:23Z
whoops, taht's a bad idea.
----
> Remove the "tickUpdate" profile config and make the "init" phase not reset
> variables
> ------------------------------------------------------------------------------------
>
> Key: METRON-668
> URL: https://issues.apache.org/jira/browse/METRON-668
> Project: Metron
> Issue Type: Improvement
> Reporter: Casey Stella
> Assignee: Casey Stella
>
> Originally during work on the MAD outlier work, I conceived of a need for a
> new callback in the profile configuration, "tickUpdate" that ran at the tick
> and had the variables from the tick that just completed available to it.
> This was done so that I could merge the state accumulated in the tick with a
> lookback window of state for MAD. The problem is that the "init" phase
> happens after this and blows away the changes done in "tickUpdate", so it
> never worked like intended.
> It occurs to me that what we really want is not to have two separate config
> phases, but only one, "init" and to not reset the variables on the tick for
> the profile. You can, of course, choose to update them by overwriting them
> in the "init" phase *or* you can choose to use them as part of your init.
> For context, this would make the example for MAD:
> {code:javascript}
> {
> "profiles": [
> {
> "profile": "sketchy_mad",
> "foreach": "'global'",
> "onlyif": "true",
> "init" : {
> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
> 'global', 5, 'MINUTES'))"
> },
> "tickUpdate": {
> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
> 'global', 5, 'MINUTES'), s)"
> },
> "update": {
> "s": "OUTLIER_MAD_ADD(s, value)"
> },
> "result": "s"
> }
> ]
> }
> {code}
> is functionally equivalent to
> {code:javascript}
> {
> "profiles": [
> {
> "profile": "sketchy_mad",
> "foreach": "'global'",
> "onlyif": "true",
> "init" : {
> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
> 'global', 5, 'MINUTES'))"
> },
> "update": {
> "s": "OUTLIER_MAD_ADD(s, value)"
> },
> "result": "s"
> }
> ]
> }
> {code}
> This resets the MAD state to the last 5 minute window. If we did NOT reset
> the state and keep accumulating state (provided we did not clear the
> variables on init, we could do the following:
> {code:javascript}
> {
> "profiles": [
> {
> "profile": "sketchy_mad",
> "foreach": "'global'",
> "onlyif": "true",
> "init" : {
> "s": "if exists(s) then s else
> OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
> 'global', 5, 'MINUTES'))"
> },
> "update": {
> "s": "OUTLIER_MAD_ADD(s, value)"
> },
> "result": "s"
> }
> ]
> }
> {code}
> s would get initialized sensibly and then always accumulate as long as the
> topology continued (rather than having a fixed lookback).
> In short, making init to not reset the variables shouldn't cause any harm and
> should provide another set of use-cases for the profiler. Also, tickUpdate
> has no function whatsoever and should be removed because it gets overwritten
> by init directly after being called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
