[
https://issues.apache.org/jira/browse/METRON-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087750#comment-16087750
]
ASF GitHub Bot commented on METRON-1005:
----------------------------------------
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/622
Your proposal has the advantage of making data in HBase self-identifying
(if one has the key), which I always like. However, it's a large change and
induces yet more complexity. There's an alternative I've been noodling
occasionally, which I put forward here for consideration:
Create a Profile Audit Log table in HBase. Every time a Profiler is
configured, started, or stopped, make one entry in the audit log. The idea is
to be able to answer exactly the kinds of questions posed in METRON-450, so the
records should include things like the configuration, the first and last
timestamps, and perhaps the key builder parameters. This would prevent
historical profiles from being "lost" because the would-be querier doesn't have
access to the exact config parameters used to write the profile.
For the sake of housekeeping, one might do a scan, daily and/or at system
restart, to assure that (a) the set Profiles with a "start" but not an "end"
recorded in the audit log, and (b) the set of currently running Profiles, are
actually consistent, and record "inferred end" entries in the audit log for
orphans found.
This solution is somewhat backward-applicable to existing Profile data; I
think there are brute-force ways to scan the existing HBase tables and infer
audit log entries, especially if historical configuration data is still
available. We could write such a scanner.
> Create Decodable Row Key for Profiler
> -------------------------------------
>
> Key: METRON-1005
> URL: https://issues.apache.org/jira/browse/METRON-1005
> Project: Metron
> Issue Type: Improvement
> Affects Versions: 0.3.0
> Reporter: Nick Allen
> Assignee: Nick Allen
> Fix For: Next + 1
>
>
> To be able to answer the types of questions that I outlined in METRON-450, we
> need a row key that is decodable. Right now there is no logic to decode a
> row key, nor is the existing row key easily decodable.
> Once the row keys can be decoded, you could scan all of the row keys in the
> Profiler's HBase table, decode each of them and extract things like, the
> names of all your profiles, the names of entities within a profile, the
> period duration of a given profile.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
