Re: Metron in-memory enrichment

I need to check it in a long run. Perhaps my test scenario was not long enough to hit the cache frequently. On Tue, Jun 20, 2017 at 12:21 AM, Ryan Merriman wrote: > "It would be nice if our enrichment cache mechanism reported hit/miss > stats or something." > > +1 to that.

Re: Metron in-memory enrichment

"It would be nice if our enrichment cache mechanism reported hit/miss stats or something." +1 to that. We do this in the JoinBolt already so it's an easy thing to add. On Mon, Jun 19, 2017 at 9:04 AM, Otto Fowler wrote: > Are you sure it is a cache miss problem? It

Re: Metron in-memory enrichment

Are you sure it is a cache miss problem? It could be that the HBASE enrichment component could just be optimized. It would be nice if our enrichment cache mechanism reported hit/miss stats or something. On June 19, 2017 at 09:58:25, Ali Nazemian (alinazem...@gmail.com) wrote: I have already

Re: Metron in-memory enrichment

I have already increased the cache value. However, clearly, the HBase enrichment is a bottleneck for the entire enrichment topology. The average latency for that part is about 30-40 ms, but the latency for other parts in total is less than 0.5 ms. On Mon, Jun 19, 2017 at 11:45 PM, Ryan Merriman

Re: Metron in-memory enrichment

The stellar enrichments do cache results. It caches at the bolt level, so it will associate an input message with computed output. On Mon, Jun 19, 2017 at 6:28 AM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > Surely the caching should make this effectively an in memory lookup.

Re: Metron in-memory enrichment

Surely the caching should make this effectively an in memory lookup. Does the stellar enrichment function not use the same clientside caching as the Hbase bolt? Simon > On 19 Jun 2017, at 06:21, Casey Stella wrote: > > In order to do that, the easiest thing to do is to

Re: Metron in-memory enrichment

That said, I think it'd be really cool to have a set of stellar functions to interact with reference data stored in MapDB (http://www.mapdb.org/) which would get localized similar to the geo enrichment stellar functions for those small-data cases. On Mon, Jun 19, 2017 at 6:21 AM, Casey Stella

Re: Metron in-memory enrichment

In order to do that, the easiest thing to do is to create a stellar function to load and do in-memory lookups. On Sun, Jun 18, 2017 at 11:48 PM, Ali Nazemian wrote: > Hi all, > > We are using Metron HBase enrichment for a few use cases, but we have > noticed the