[
https://issues.apache.org/jira/browse/METRON-701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854407#comment-15854407
]
Nick Allen commented on METRON-701:
-----------------------------------
There are multiple ways to consider to implementing a mechanism to write
profile measurements to Kafka. Here are two to consider.
(1) Alter the Profile Definitions
The Profile author would choose if their profile data needs to be re-ingested
as telemetry. The author would use a Stellar function to write the profile
measurement to Kafka when the profile is flushed. This would expand the
profile definition to accommodate this new feature.
Pros
* Simple - Simpler to implement
* Flexibility - Profile data that is not triaged, does not need to be
re-ingested and processed
Cons
* Difficult to Scale/Optimize - It will be difficult to optimize the writes to
Kafka. An increase in either the number of profiles, or the number of entities
being profiled (which can occur without any change to the Profiler itself)
could cause scaling problems that cannot be easily addressed.
* More Complex Profile Definition - Makes the Profile definition more complex
and forces an additional decision on the user.
* More Complex Triage - Introduces uncertainty about which profiles will be
available for triage. Adding a new triage may require updating the threat
triage rules AND updating the Profile definition.
(2) Alter the Profiler Topology
Alter the Profiler Topology, so that when a profile is flushed the measurement
is sent to a downstream bolt for writing to HBase and another downstream bolt
for writing to Kafka. There would be a global switch to write to Kafka or not.
Either all Profiles write to Kafka or none do.
Pros
* Easier to Scale/Optimize - Provides a well-understood mechanism to optimize
the writes to Kafka. Scaling the writes to Kafka could be done in the same
manner as done for HBase.
* Less Complex Profile Definition
* Less Complex Triage - Less complex for the user as their is no uncertainty
about which Profile data is available for threat triage.
Cons
* Less Flexible - Less flexible as either all Profiles write to Kafka or none
do.
> Triage Metrics Produced by the Profiler
> ---------------------------------------
>
> Key: METRON-701
> URL: https://issues.apache.org/jira/browse/METRON-701
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> h3. Problem
> The motivating example is that I would like to create an alert if the number
> of inbound flows to any host over a 15 minute interval is abnormal.
> The value being interrogated here, the number of inbound flows, is not a
> static value contained within any single telemetry message. This value is
> calculated across multiple messages by the Profiler. The current Threat
> Triage process cannot be used to interrogate values calculated by the
> Profiler.
> h3. Proposed Solution
> I am proposing that we treat the Profiler as a source of telemetry. The
> measurements captured by the Profiler would be enqueued into a Kafka topic.
> We would then treat those Profiler messages like any other telemetry. We
> would parse, enrich, triage, and index those messages.
> This would have the following advantages.
> 1. We would be able to reuse the same threat triage mechanism for values
> calculated by the Profiler.
> 2. We would be able to generate profiles from the profiled data - aka
> meta-profiles anyone?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
