I need to check it in a long run. Perhaps my test scenario was not long enough to hit the cache frequently.
On Tue, Jun 20, 2017 at 12:21 AM, Ryan Merriman <[email protected]> wrote: > "It would be nice if our enrichment cache mechanism reported hit/miss > stats or something." > > +1 to that. We do this in the JoinBolt already so it's an easy thing to > add. > > On Mon, Jun 19, 2017 at 9:04 AM, Otto Fowler <[email protected]> > wrote: > >> Are you sure it is a cache miss problem? It could be that the HBASE >> enrichment component could just be optimized. >> It would be nice if our enrichment cache mechanism reported hit/miss >> stats or something. >> >> >> On June 19, 2017 at 09:58:25, Ali Nazemian ([email protected]) wrote: >> >> I have already increased the cache value. However, clearly, the HBase >> enrichment is a bottleneck for the entire enrichment topology. The average >> latency for that part is about 30-40 ms, but the latency for other parts in >> total is less than 0.5 ms. >> >> On Mon, Jun 19, 2017 at 11:45 PM, Ryan Merriman <[email protected]> >> wrote: >> >>> We already front HBase enrichments with a cache (actually all >>> enrichments have a cache in front, see the GenericEnrichmentBolt class). >>> You probably just need to increase the cache size. This setting is in the >>> enrichment flux file at $METRON_HOME/flux/enrichment/remote.yaml. Look >>> for the "withMaxCacheSize" setting under the appropriate bolt. >>> >>> Ryan >>> >>> On Mon, Jun 19, 2017 at 8:31 AM, Casey Stella <[email protected]> >>> wrote: >>> >>>> The stellar enrichments do cache results. It caches at the bolt level, >>>> so it will associate an input message with computed output. >>>> >>>> On Mon, Jun 19, 2017 at 6:28 AM, Simon Elliston Ball < >>>> [email protected]> wrote: >>>> >>>>> Surely the caching should make this effectively an in memory lookup. >>>>> Does the stellar enrichment function not use the same clientside caching >>>>> as >>>>> the Hbase bolt? >>>>> >>>>> Simon >>>>> >>>>> On 19 Jun 2017, at 06:21, Casey Stella <[email protected]> wrote: >>>>> >>>>> In order to do that, the easiest thing to do is to create a stellar >>>>> function to load and do in-memory lookups. >>>>> >>>>> On Sun, Jun 18, 2017 at 11:48 PM, Ali Nazemian <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> We are using Metron HBase enrichment for a few use cases, but we have >>>>>> noticed the achievable throughput is not very great. I was wondering >>>>>> whether there is a way to load the external enrichment data in-memory and >>>>>> use it with normal Stellar enrichments. In our use cases, the number of >>>>>> rows in the external enrichments that we are dealing with is less than a >>>>>> 100k and it is a static list, so it is feasible to load them in-memory >>>>>> and >>>>>> use that for the enrichment. However, I am not sure how that would be >>>>>> achievable from the Metron capabilities. >>>>>> >>>>>> Regards, >>>>>> Ali >>>>>> >>>>> >>>>> >>>> >>> >> >> >> -- >> A.Nazemian >> >> > -- A.Nazemian
