We already front HBase enrichments with a cache (actually all enrichments have a cache in front, see the GenericEnrichmentBolt class). You probably just need to increase the cache size. This setting is in the enrichment flux file at $METRON_HOME/flux/enrichment/remote.yaml. Look for the "withMaxCacheSize" setting under the appropriate bolt.
Ryan On Mon, Jun 19, 2017 at 8:31 AM, Casey Stella <[email protected]> wrote: > The stellar enrichments do cache results. It caches at the bolt level, so > it will associate an input message with computed output. > > On Mon, Jun 19, 2017 at 6:28 AM, Simon Elliston Ball < > [email protected]> wrote: > >> Surely the caching should make this effectively an in memory lookup. Does >> the stellar enrichment function not use the same clientside caching as the >> Hbase bolt? >> >> Simon >> >> On 19 Jun 2017, at 06:21, Casey Stella <[email protected]> wrote: >> >> In order to do that, the easiest thing to do is to create a stellar >> function to load and do in-memory lookups. >> >> On Sun, Jun 18, 2017 at 11:48 PM, Ali Nazemian <[email protected]> >> wrote: >> >>> Hi all, >>> >>> We are using Metron HBase enrichment for a few use cases, but we have >>> noticed the achievable throughput is not very great. I was wondering >>> whether there is a way to load the external enrichment data in-memory and >>> use it with normal Stellar enrichments. In our use cases, the number of >>> rows in the external enrichments that we are dealing with is less than a >>> 100k and it is a static list, so it is feasible to load them in-memory and >>> use that for the enrichment. However, I am not sure how that would be >>> achievable from the Metron capabilities. >>> >>> Regards, >>> Ali >>> >> >> >
