I think setting TTL values for enrichments should be fine for the time being.
I am not sure what the issue was originally where we didn't feel it was
sufficient (probably performance implication on Hbase?), but I think it should
be safe reverting back to setting TTL.
James
13.08.2019, 15:44, "Casey Stella" <ceste...@gmail.com>:
> Ah, that feature. Yes, it never seemed to catch on. It actually wasn't
> from OpenSOC, but a very early feature of Metron. The use-case was that
> enrichments may go stale and removing them based on TTL was easy to do, but
> not ideal. The LeastRecentlyUsedPruner was a MR job which would allow
> enrichments to be pruned which had not been *read* in x amount of time. It
> did this by capturing bloom filters with enrichment keys used for a
> time-range and the MR job would use those bloom filters to determine which
> keys to remove.
>
> I'd be ok with it either being used or removed. It's unclear to me whether
> the use-case that hbase needs to be pruned based on usage was as valid as
> we thought. I guess that makes me +0 on the request to deprecate.
>
> On Tue, Aug 13, 2019 at 6:28 PM Nick Allen <n...@nickallen.org> wrote:
>
>> Sure. I should have provided some more context. I can tell you what I do
>> know about it. Perhaps others can provide some more color.
>>
>> - This is functionality accessed by a user by running the script; ${
>> METRON_HOME}/bin/threatintel_bulk_prune.sh
>>
>> - If you are using access trackers with your HBase enrichments, it runs
>> as an MR job that counts the number of times each Enrichment is used.
>> I am
>> assuming that it then prunes those that are less frequently accessed.
>>
>> - It was originally created here;
>> https://github.com/apache/metron/pull/22
>>
>> On Tue, Aug 13, 2019 at 6:11 PM Otto Fowler <ottobackwa...@gmail.com>
>> wrote:
>>
>> > Can you summarize what it does? Is it from OpenSOC?
>> >
>> >
>> >
>> >
>> > On August 13, 2019 at 17:53:52, Nick Allen (n...@nickallen.org) wrote:
>> >
>> > As part of https://github.com/apache/metron/pull/1470, I found it
>> > difficult
>> > to update the "Least Recently Used Pruner" to work with HBase 2.0.2. I am
>> > sure that given more time and effort, I could make it work, but is it
>> worth
>> > it?
>> >
>> > This is a feature that I myself am not familiar with. I do not know of
>> > anyone using this. I also did not find much documentation on how to use
>> > this feature. I certainly don't know the entire user community, so please
>> > let me know if anyone is using this functionality or believes that it
>> > should be maintained going forward.
>> >
>> > Would you support deprecating this feature?
>> >
>> > Thanks
>> >
-------------------
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
