"It would be nice if our enrichment cache mechanism reported hit/miss stats or something."
+1 to that. We do this in the JoinBolt already so it's an easy thing to add. On Mon, Jun 19, 2017 at 9:04 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > Are you sure it is a cache miss problem? It could be that the HBASE > enrichment component could just be optimized. > It would be nice if our enrichment cache mechanism reported hit/miss stats > or something. > > > On June 19, 2017 at 09:58:25, Ali Nazemian (alinazem...@gmail.com) wrote: > > I have already increased the cache value. However, clearly, the HBase > enrichment is a bottleneck for the entire enrichment topology. The average > latency for that part is about 30-40 ms, but the latency for other parts in > total is less than 0.5 ms. > > On Mon, Jun 19, 2017 at 11:45 PM, Ryan Merriman <merrim...@gmail.com> > wrote: > >> We already front HBase enrichments with a cache (actually all enrichments >> have a cache in front, see the GenericEnrichmentBolt class). You probably >> just need to increase the cache size. This setting is in the enrichment >> flux file at $METRON_HOME/flux/enrichment/remote.yaml. Look for the >> "withMaxCacheSize" setting under the appropriate bolt. >> >> Ryan >> >> On Mon, Jun 19, 2017 at 8:31 AM, Casey Stella <ceste...@gmail.com> wrote: >> >>> The stellar enrichments do cache results. It caches at the bolt level, >>> so it will associate an input message with computed output. >>> >>> On Mon, Jun 19, 2017 at 6:28 AM, Simon Elliston Ball < >>> si...@simonellistonball.com> wrote: >>> >>>> Surely the caching should make this effectively an in memory lookup. >>>> Does the stellar enrichment function not use the same clientside caching as >>>> the Hbase bolt? >>>> >>>> Simon >>>> >>>> On 19 Jun 2017, at 06:21, Casey Stella <ceste...@gmail.com> wrote: >>>> >>>> In order to do that, the easiest thing to do is to create a stellar >>>> function to load and do in-memory lookups. >>>> >>>> On Sun, Jun 18, 2017 at 11:48 PM, Ali Nazemian <alinazem...@gmail.com> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> We are using Metron HBase enrichment for a few use cases, but we have >>>>> noticed the achievable throughput is not very great. I was wondering >>>>> whether there is a way to load the external enrichment data in-memory and >>>>> use it with normal Stellar enrichments. In our use cases, the number of >>>>> rows in the external enrichments that we are dealing with is less than a >>>>> 100k and it is a static list, so it is feasible to load them in-memory and >>>>> use that for the enrichment. However, I am not sure how that would be >>>>> achievable from the Metron capabilities. >>>>> >>>>> Regards, >>>>> Ali >>>>> >>>> >>>> >>> >> > > > -- > A.Nazemian > >
