This is the "enrichment.join.cache.size" setting in
$METRON_HOME/flux/enrichment/remote.yaml. Depending on the version you're
on, that property may not have made it into the flux file yet. In that
case, look for this section in the flux file:
- id: "enrichmentJoinBolt"
className: "org.apache.metron.enrichment.bolt.EnrichmentJoinBolt"
constructorArgs:
- "${kafka.zk}"
configMethods:
- name: "withMaxCacheSize"
args: [${enrichment.join.cache.size}]
- name: "withMaxTimeRetain"
args: [10]
This property is available in $METRON_HOME/config/enrichment.properties and
Ambari in master. There is also a matching threat intel setting so look
there if this doesn't fix the error. The stacktrace doesn't tell you which
one it is but it's probably the enrichment join bolt since it comes before
threat intel.
Ryan
On Fri, Aug 4, 2017 at 12:44 PM, Laurens Vets <[email protected]> wrote:
> Hi list,
>
> I see the following error in my enrichmentJoinBolt Storm UI:
>
> java.lang.Exception: Join cache reached max size limit. Increase the
> maxCacheSize setting or add more tasks to enrichment/threatintel join bolt.
> at
> org.apache.metron.enrichment.bolt.JoinBolt$JoinRemoveListener.onRemoval(JoinBolt.java:107)
> at org.apache.metron.guava.enrichment.cache.LocalCache.processP
> endingNotifications(LocalCache.java:2011) at
> org.apache.metron.guava.enrichment.cache.LocalCache$Segment.
> runUnlockedCleanup(LocalCache.java:3501) at org.apache.metron.guava.enrich
> ment.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3477) at
> org.apache.metron.guava.enrichment.cache.LocalCache$Segment.
> storeLoadedValue(LocalCache.java:3208) at org.apache.metron.guava.enrich
> ment.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2408) at
> org.apache.metron.guava.enrichment.cache.LocalCache$Segment.loadSync(LocalCache.java:2373)
> at org.apache.metron.guava.enrichment.cache.LocalCache$Segment.
> lockedGetOrLoad(LocalCache.java:2335) at org.apache.metron.guava.enrich
> ment.cache.LocalCache$Segment.get(LocalCache.java:2250) at
> org.apache.metron.guava.enrichment.cache.LocalCache.get(LocalCache.java:3980)
> at
> org.apache.metron.guava.enrichment.cache.LocalCache.getOrLoad(LocalCache.java:3984)
> at org.apache.metron.guava.enrichment.cache.LocalCache$LocalLoa
> dingCache.get(LocalCache.java:4868) at org.apache.metron.enrichment.b
> olt.JoinBolt.execute(JoinBolt.java:129) at org.apache.storm.daemon.execut
> or$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466)
> at
> org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40)
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> at
> org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853)
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) at
> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
>
> How can I increase this cache?
>
