GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/395
METRON-590 Enable Use of Event Time in Profiler
## [METRON-590](https://issues.apache.org/jira/browse/METRON-590)
### Changes
* Added event time processing support to the Profiler. Previously the
Profiler only supported processing time aka wall clock time processing. Event
time processing is advantageous as it is not susceptible to skew caused by
heavy processing load, allows the reprocessing/replay of archived telemetry
data, and under certain circumstances can produce a more accurate profile of
entity behavior.
* By default, the Profiler will use event time processing. The Flux
topology definition file must be edited to switch the Profiler to wall clock or
processing time.
* The Profiler is now leveraging Storm's windowing functionality introduced
in Storm 1.x. This provides the core engine for event time processing. This
also provides a means for the use of different window types, like sliding
windows, in the Profiler. This is currently not exposed to users of the
Profiler as the Flux topology definition file must be edited to use different
window types.
* Enhanced the Profiler integration tests which was enabled by the use of
event time processing. The integration tests now generate 24 hours of
telemetry data at roughly 3 messages per minute, and then flush profile values
every 15 minutes. The entire stream of values generated by the Profiler is
then validated for correctness.
* Created a `ConfigurationManager` that can be used to read the latest
configuration changes in a remote data store like Zookeeper. The default
implementation, `ZkConfigurationManager` replicates the functionality that is
embedded in the `ConfiguredBolt` base class. The Profiler bolts can no longer
subclass `ConfiguredBolt` as it subclasses Storm's `BaseRichBolt` which will
not work for the Profiler bolts.
* The usability of the Profiler was enhanced to better support active
profiles that are subsequently edited by the user. Changes should be handled
seamlessly by the Profiler. This is especially helpful when a mistake is made
when creating a profile, which then needs to be fixed and updated. The
Profiler was also made more resilient to failures specific to a single Profile
or Tuple. Individual failures should not impact other Profiles or Tuples.
### Testing
Tested on a multi-node AWS cluster and the Quick Dev environment. Created,
edited, and deleted multiple profile definitions as the Profiler was running
and responding to the changes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-590
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/395.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #395
----
commit cca756b7781ee7058edadfa84777bce7286d7817
Author: Nick Allen <[email protected]>
Date: 2016-12-07T20:14:07Z
METRON-590 Enable Use of Event Time in Profiler
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
