Hi all,
We are writing an Elasticsearch appender and were assuming there would be one object instance per Appender but log4j2 seems to be initializing a lot more than one instance of the Provider class (via plugin). I'm assuming we are missing something in terms of the Appender lifecycle, and hoping to find some help here. Technicalities: 1. Our Appender plugin is here: https://github.com/BigDataBoutique/log4j2-elasticsearch-http. It is based on the NoSQL Appender and consists of one Provider class (https://github.com/BigDataBoutique/log4j2-elasticsearch-http/blob/master/src/main/java/com/bigdataboutique/logging/log4j2/ElasticsearchHttpProvider.java) and one HttpClient tasks which batches the requests to Elasticsearch (https://github.com/BigDataBoutique/log4j2-elasticsearch-http/blob/master/src/main/java/com/bigdataboutique/logging/log4j2/ElasticsearchHttpClient.java). 2. The entire idea behind this Appender is to isolate network / cluster failures from log producing software, so in case of any type of failure some messages will only accumulate some memory and nothing more than that. Adding new logs from the actual using software should be either cheap or no-op. 3. We added a simple log to show us how many Provider (and hence HttpClient) instances are created. We see many of those lines when the app is starting and many after (example: https://pastebin.com/RRbDw0T9). 4. This basically means the Provider instance is being created for each logger, probably on each thread (?) and is not being reused. This, in turn, means we have many HttpClient instances perfoming flushes to a remote Elasticsearch cluster essentially bombarding it with HTTP requests (instead of batching them together and periodically executing that). Questions: 1. Is that behavior expected? documented anywhere? 2. Can this be avoided, and log4j2 be told to reuse the provider instance? 3. Would it make sense to cache the Provider instance when we detect the same "connection string" is used and return the existing instance instead of creating a new one? or log4j internally dismisses appenders and could dispose one we cached on the Provider class level? 4. Docs on NoSqlProvider::getConnection seem to suggest caching is recommended on that method so the underlying connection can be reused. Can anyone confirm? and how does that differentiate from caching the Provider object, which seems to be created many times more than it needs to? We are currently using 2.8.1 and the entire source code is available for your review (see links above). Will appreciate any help on this! Thanks, Itamar.
