Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1099#discussion_r202785248
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -182,40 +185,61 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
super.prepare(stormConf, context, collector);
messageGetStrategy = MessageGetters.DEFAULT_BYTES_FROM_POSITION.get();
this.collector = collector;
- if(getSensorParserConfig() != null) {
- cache =
CachingStellarProcessor.createCache(getSensorParserConfig().getCacheConfig());
- }
- initializeStellar();
- if(getSensorParserConfig() != null && filter == null) {
-
getSensorParserConfig().getParserConfig().putIfAbsent("stellarContext",
stellarContext);
- if
(!StringUtils.isEmpty(getSensorParserConfig().getFilterClassName())) {
- filter = Filters.get(getSensorParserConfig().getFilterClassName()
- , getSensorParserConfig().getParserConfig()
- );
+
+ // Build the Stellar cache
+ Map<String, Object> cacheConfig = new HashMap<>();
+ for (Map.Entry<String, ParserComponents> entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ SensorParserConfig config = getSensorParserConfig(sensor);
+
+ if (config != null) {
+ cacheConfig.putAll(config.getCacheConfig());
}
}
+ cache = CachingStellarProcessor.createCache(cacheConfig);
- parser.init();
+ // Need to prep all sensors
+ for (Map.Entry<String, ParserComponents> entry:
sensorToComponentMap.entrySet()) {
+ String sensor = entry.getKey();
+ MessageParser<JSONObject> parser =
entry.getValue().getMessageParser();
--- End diff --
I left it as a single shared cache on purpose.
I don't believe that there'd be any incorrect evictions by sharing the
cache, and I think evicting based on the overall usage in the aggregated parser
is the appropriate place to handle it. Since the cache is (mostly) LRU, I'd
prefer to drop the least recently used entry of all parsers rather than
dropping for each parser. LRU of the overall flow seems better than LRU of
each of the sensors. Assuming a single cache, you'd bump up the cache configs
to account for this, rather than having to optimize each config individually
(and potentially as a group afterwards).
Caching is also off by default for the parsers, so this is a case that's
only hit if the user explicitly chooses to do so.
Having said that, I do think I need to shore up the documentation around
that logic, assuming we choose to go forward with it. Let me know what you
think, and I can adjust appropriately.
---
