[
https://issues.apache.org/jira/browse/METRON-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15765057#comment-15765057
]
ASF GitHub Bot commented on METRON-590:
---------------------------------------
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/395#discussion_r93309115
--- Diff: metron-analytics/metron-profiler-client/README.md ---
@@ -86,4 +71,73 @@ Returns: The profile measurements.
[12078.0, 8921.0, 12131.0]
```
-The client API call above has retrieved the past hour of the 'test'
profile for the entity '192.168.138.158'.
\ No newline at end of file
+The client API call above has retrieved the past hour of the 'test'
profile for the entity '192.168.138.158'.
+
+### Configuration
+
+By default, the Profiler creates Profiles with a period duration of 15
minutes. This means that data is accumulated, summarized and flushed every 15
minutes. The Client API must also have knowledge of this duration to correctly
retrieve the profile data. If the client API is expecting 15 minute periods, it
will not be able to read data generated by a Profiler that has been configured
with a 1 hour period.
+
+The period duration can be configured in the Profiler by altering the
Profiler topology's static properties file
(`$METRON/config/profiler.properties`). The Stellar Client API can be
configured by setting the following properties in Metron's global configuration.
+
+| Key | Description
| Required | Default |
+| ------------------------------------- | -------- | -------- | -------- |
+| profiler.client.period.duration | The duration of each profile
period. This value should be defined along with
`profiler.client.period.duration.units`. | Optional | 15 |
+| profiler.client.period.duration.units | The units used to specify the
profile period duration. This value should be defined along with
`profiler.client.period.duration`. | Optional | MINUTES |
+| profiler.client.hbase.table | The name of the HBase table used
to store profile data. | Optional | profiler |
+| profiler.client.hbase.column.family | The name of the HBase
column family used to store profile data. | Optional | P |
+| profiler.client.salt.divisor | The salt divisor used to store
profile data. | Optional | 1000 |
+| hbase.provider.impl | The name of the
HBaseTableProvider implementation class. | Optional | |
+
+### Functions
+
+#### `PROFILE_GET`
+
+Retrieves a series of values from a stored profile.
+
+Arguments
+ * profile - The name of the profile.
+ * entity - The name of the entity.
+ * durationAgo - How long ago should values be retrieved from?
+ * units - The units of 'durationAgo'.
+ * groups - Optional - The groups used to sort the profile.
+
+Returns
+ * The profile measurements.
+
+Examples
+
+Retrieve the last 30 minutes of measurements for profile 'profile1' and
entity '10.0.0.1'.
+```
+PROFILE_GET('profile1', '10.0.0.1', 30, 'MINUTES')
+```
+
+Retrieve the last 30 minutes of measurements for profile 'profile1' and
entity '10.0.0.1' in the group named 'mondays'.
+```
+PROFILE_GET('profile1', '10.0.0.1', 30, 'MINUTES', 'mondays')
+```
+
+#### `PROFILE_GET_FROM`
+
+Retrieves a series of values from a stored profile.
+
+Arguments
+ * profile - The name of the profile.
+ * entity - The name of the entity.
+ * lookBack - How long to look back in milliseconds
+ * offset - Optional - When to start the look back from in epoch
milliseconds. Defaults to current time.
+ * groups - Optional - The groups used to sort the profile.
--- End diff --
I also got into this issue in
[PR-345](https://github.com/apache/incubator-metron/pull/345) where I added
"Profile Period" as an optional argument to PROFILE_GET. As noted there, this
is also mostly for the benefit of the Replay use case. This resulted in two
optional arguments. During parsing I used the argument type to distinguish
which if either were being provided, which worked and was easy, but certainly
felt like a hack.
I think @cestella 's comment about using a final Map for all (named)
optional arguments is very insightful. It gives us named arguments without an
actual syntax extension, and allows multiple optional args. I think we should
proceed with that approach, at least for PROFILE_GET_FROM.
> Enable Use of Event Time in Profiler
> ------------------------------------
>
> Key: METRON-590
> URL: https://issues.apache.org/jira/browse/METRON-590
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> There are at least two different times that are important to consider when
> handling the telemetry messages received by Metron.
> (1) Processing time is the time at which Metron processed the message.
> (2) Event time is the time at which the event actually occurred.
> If Metron is consuming live data and all is well, the processing and event
> times may remain close and consistent. When processing time differs from
> event time the data produced by the Profiler may be inaccurate. There are a
> few scenarios under which these times might differ greatly which would
> negatively impact the feature set produced by the Profiler.
> (1) When the system has experienced an outage, for example, a scheduled
> maintenance window. When restarted a high volume of messages will need to be
> processed by the Profiler. The output of the Profiler will indicate an
> increase in activity, although no change in activity actually occurred on the
> target network. This could happen whether the outage was Metron itself or an
> upstream system that feeds data to Metron.
> (2) If the user attempts to replay historical telemetry through the Profiler,
> the Profiler will attribute the activity to the time period in which it was
> processed. Obviously the activity should be attributed to the time period in
> which the raw telemetry events originated in.
> There are some scenarios when processing time might be preferred and other
> use cases where event time is preferred. The Profiler should be enhanced to
> allow it to produce profiles based on either processing time or event time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
