[
https://issues.apache.org/jira/browse/METRON-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767146#comment-15767146
]
ASF GitHub Bot commented on METRON-590:
---------------------------------------
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/395
@mattf-horton @mmiklavc First off, thanks for the perspective, both of
you. While I think we should investigate your suggestion, @mattf-horton , it
appears on the face of it quite dependent upon implementation decisions and
details inside of Storm. That immediately gives me pause. That being said,
I'll repeat that I think we should consider it strongly.
I do agree with @mattf-horton that pulling out the `ZkConfigurationManager`
here would be substantially more work if you consider the impact to our tests
and the increased amount of testing that we would have to add to this PR. I
think, as @mmiklavc suggested, it's a separate PR. I would prefer to get this
really good work by @nickwallen in, so maybe I can offer another compromise
position to go with @mattf-horton 's one earlier.
I'd like to propose another compromise. The main concern that I have is
leaving an second abstraction in place for managing configuration from
zookeeper. Given that, what do you think of the following:
* Remove the `ConfigurationManager` abstraction from this PR
* Since it is used only in one class now, directly use the
`org.apache.curator.framework.recipes.cache.NodeCache` in `ProfilerSplitterBolt`
* Create a follow-on JIRA and make a comment with a TODO suggesting we
replace this with the appropriate abstraction (referencing the JIRA).
This would remove the possibility of a fork and be substantially less
work/impact.
Thoughts?
> Enable Use of Event Time in Profiler
> ------------------------------------
>
> Key: METRON-590
> URL: https://issues.apache.org/jira/browse/METRON-590
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> There are at least two different times that are important to consider when
> handling the telemetry messages received by Metron.
> (1) Processing time is the time at which Metron processed the message.
> (2) Event time is the time at which the event actually occurred.
> If Metron is consuming live data and all is well, the processing and event
> times may remain close and consistent. When processing time differs from
> event time the data produced by the Profiler may be inaccurate. There are a
> few scenarios under which these times might differ greatly which would
> negatively impact the feature set produced by the Profiler.
> (1) When the system has experienced an outage, for example, a scheduled
> maintenance window. When restarted a high volume of messages will need to be
> processed by the Profiler. The output of the Profiler will indicate an
> increase in activity, although no change in activity actually occurred on the
> target network. This could happen whether the outage was Metron itself or an
> upstream system that feeds data to Metron.
> (2) If the user attempts to replay historical telemetry through the Profiler,
> the Profiler will attribute the activity to the time period in which it was
> processed. Obviously the activity should be attributed to the time period in
> which the raw telemetry events originated in.
> There are some scenarios when processing time might be preferred and other
> use cases where event time is preferred. The Profiler should be enhanced to
> allow it to produce profiles based on either processing time or event time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
