[
https://issues.apache.org/jira/browse/METRON-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762614#comment-15762614
]
ASF GitHub Bot commented on METRON-590:
---------------------------------------
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/395#discussion_r93142265
--- Diff:
metron-analytics/metron-profiler/src/main/config/profiler.properties ---
@@ -18,21 +18,44 @@
#
#
-##### Storm #####
+##### Profiler #####
-profiler.workers=1
-profiler.executors=0
+# topic from which the profiler consumes messages
profiler.input.topic=indexing
+
+# how frequently the profiler summarizes data - aka duration of each
profile period
profiler.period.duration=15
profiler.period.duration.units=MINUTES
+
+# lifespan of a profile - must be greater than twice the profile period
+# intermediate state may be lost if no messages received within this time
span
profiler.ttl=30
profiler.ttl.units=MINUTES
-profiler.hbase.salt.divisor=1000
+
+# event time processing - the name of the field containing the event
timestamp
+profiler.event.timestamp.field=timestamp
+
+# event time processing - how long time lags behind the last seen timestamp
+profiler.event.time.lag=1
+profiler.event.time.lag.units=MINUTES
+
+# where profiles are written to in hbase
profiler.hbase.table=profiler
profiler.hbase.column.family=P
+
+# how profiles are written to hbase
+profiler.hbase.salt.divisor=1000
--- End diff --
Perhaps a stupid question, but wouldn't we want the salt divisor in
zookeeper so the reader and the writer are both sync'd? Profiles written with
salt divisors of `x` can only be read by clients that use salt divisors of `x`,
right?
> Enable Use of Event Time in Profiler
> ------------------------------------
>
> Key: METRON-590
> URL: https://issues.apache.org/jira/browse/METRON-590
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> There are at least two different times that are important to consider when
> handling the telemetry messages received by Metron.
> (1) Processing time is the time at which Metron processed the message.
> (2) Event time is the time at which the event actually occurred.
> If Metron is consuming live data and all is well, the processing and event
> times may remain close and consistent. When processing time differs from
> event time the data produced by the Profiler may be inaccurate. There are a
> few scenarios under which these times might differ greatly which would
> negatively impact the feature set produced by the Profiler.
> (1) When the system has experienced an outage, for example, a scheduled
> maintenance window. When restarted a high volume of messages will need to be
> processed by the Profiler. The output of the Profiler will indicate an
> increase in activity, although no change in activity actually occurred on the
> target network. This could happen whether the outage was Metron itself or an
> upstream system that feeds data to Metron.
> (2) If the user attempts to replay historical telemetry through the Profiler,
> the Profiler will attribute the activity to the time period in which it was
> processed. Obviously the activity should be attributed to the time period in
> which the raw telemetry events originated in.
> There are some scenarios when processing time might be preferred and other
> use cases where event time is preferred. The Profiler should be enhanced to
> allow it to produce profiles based on either processing time or event time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
