[
https://issues.apache.org/jira/browse/METRON-529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15629580#comment-15629580
]
ASF GitHub Bot commented on METRON-529:
---------------------------------------
Github user mattf-horton commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/334#discussion_r86190240
--- Diff:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
---
@@ -226,14 +262,23 @@ private ColumnBuilder getColumnBuilder(Map<String,
Object> global) {
* @param global The global configuration.
*/
private RowKeyBuilder getRowKeyBuilder(Map<String, Object> global) {
- /*
- * WARNING: the row key builder is not currently configurable. by
invoking
- * the default constructor below, this defaults to generating keys
using a
- * period duration of 15 minutes. this function will NOT be able to
read
- * profiles created by a profiler running with any other period
duration, but
- * 15 minutes.
- */
- return new SaltyRowKeyBuilder();
+
+ // how long is the profile period?
+ String configuredDuration = (String)
global.getOrDefault(PROFILER_PERIOD, PROFILER_PERIOD_DEFAULT);
+ long duration = Long.parseLong(configuredDuration);
+ LOG.debug("profiler client: {}={}", PROFILER_PERIOD, duration);
+
+ // which units are used to define the profile period?
+ String configuredUnits = (String)
global.getOrDefault(PROFILER_PERIOD_UNITS, PROFILER_PERIOD_UNITS_DEFAULT);
+ TimeUnit units = TimeUnit.valueOf(configuredUnits);
+ LOG.debug("profiler client: {}={}", PROFILER_PERIOD_UNITS, units);
+
+ // what is the salt divisor?
+ String configuredSaltDivisor = (String)
global.getOrDefault(PROFILER_SALT_DIVISOR, PROFILER_SALT_DIVISOR_DEFAULT);
+ int saltDivisor = Integer.parseInt(configuredSaltDivisor);
+ LOG.debug("profiler client: {}={}", PROFILER_SALT_DIVISOR,
saltDivisor);
+
+ return new SaltyRowKeyBuilder(saltDivisor, duration, units);
--- End diff --
I had thought about the profiler having that limitation, I'm glad you
opened the jira, and I definitely agree that can be deferred. Regarding the
client, though, I'm thinking about this scenario:
(a) There are one or two clients doing on-going routine analysis of
up-to-date profiles, perhaps they are updaters for ML modules;
(b) Suddenly a security event requires analyzing some historical profile
data from 2 months ago, that was done with different profiles -- different
table, different period, something.
We shouldn't need to turn off (a) in order to do (b).
I won't insist on this, because certainly what you've already done is a
great improvement. But I'd still like to suggest that a small addition to the
mods you've made in GetProfile would give this flexibility. Do you mind if I
just whomp it up? Then you can look at it, and if you still don't like it,
I'll shut up :-)
> Allow Configurable Periods for Profiler Client
> ----------------------------------------------
>
> Key: METRON-529
> URL: https://issues.apache.org/jira/browse/METRON-529
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> By default, the Profiler creates Profiles with a period duration of 15
> minutes. This means that data is accumulated, summarized and flushed every 15
> minutes. The Client API must also have knowledge of this duration to
> correctly retrieve the profile data. If the client API is expecting 15 minute
> periods, it will not be able to read data generated by a Profiler that has
> been configured with a 1 hour period.
> The period duration can be configured in the Profiler by altering the
> Profiler topology's static properties file. The Stellar Client API currently
> provides no means to configure the period duration and defaults also to 15
> minutes. This means that the Stellar Client API can only read profiles with a
> period duration of 15 minutes. This is a known limitation that has not yet
> been addressed.
> https://github.com/apache/incubator-metron/blob/e31705029fd7518bd03c22eece0c2a8eb1187abf/metron-analytics/metron-profiler-client/README.md#period-duration
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
