The AbstractIpcAppender thing is more of a general issue that doesn't really need to be solved at this time. It's just a thought I had while thinking that SQL and NoSQL are not the only style appenders that can benefit. This could eventually be useful in refactoring the MOM appenders (JMS, Kafka, ZeroMQ, etc).
As for the fast reply, you lucked out while I was online. ;) Since Jedis doesn't have a failover mode like Cassandra, Kafka, ActiveMQ, etc., all do, then you may want to investigate which approach makes more sense: adding the feature within the appender, relying on the failover appender, or something else entirely. On 6 May 2017 at 14:06, Volkan Yazıcı <[email protected]> wrote: > Appreciated the detailed prompt reply, thanks Matt! > > 1. Then I will go with Jedis without any Maven shade plugin hacks. > 2. BurstFilter covers rate limiting. > 3. AsyncLogger covers consumer-producer rate mismatch regulation. > 4. I've checked the CassandraAppender. It leverages the multiple contact > point feature provided by the Cassandra client. I'm thinking of doing > this > manually, since none offered by Jedis. > 5. I will employ AbstractDatabaseAppender for batching. > 6. I will tackle AbstractDatabaseAppender->AbstractIpcAppender problem > later on. > > I will create a JIRA ticket right now with these goals. > > On Sat, May 6, 2017 at 8:49 PM, Matt Sicker <[email protected]> wrote: > > > Instead of forming a new thread to point this out, I'll note it here: the > > AbstractDatabaseAppender classes may be refactorable or renamable into a > > sort of AbstractIpcAppender instead as another major category of > appenders. > > Might be worth examining, though it's something we'd help in structuring > at > > least for other appenders and whatnot. If you have any questions on best > > practices, let us know! > > > > Also, it would be great to submit a JIRA issue for this as well so we > have > > somewhere to track the change. > > > > On 6 May 2017 at 13:45, Matt Sicker <[email protected]> wrote: > > > > > Oh, I also forgot to mention that regarding batching of log messages, > > > several appenders already follow a similar pattern, so you may find > some > > > reusable code there, especially if you're modeling this from the > > > NoSqlAppender/Manager abstract classes as well as the > > > AbstractDatabaseAppender/Manager classes which are beginning to sound > > > like they have a too-specific name. :) > > > > > > On 6 May 2017 at 13:43, Matt Sicker <[email protected]> wrote: > > > > > >> First of all, we love contributions, and this one sounds interesting, > > too. > > >> > > >> 1. I don't have experience with Redis (I've used other distributed > > memory > > >> caches and data grid frameworks, though), so I don't have an opinion > on > > >> which library to use for it. > > >> > > >> 2. As for the RateLimiter class, I'll note that we have a BurstFilter > < > > >> https://logging.apache.org/log4j/2.x/manual/filters.html#BurstFilter> > > >> which sounds similar to what you're aiming for. In general, rate > > limiting > > >> log messages would be more of a Filter concern than an Appender > > concern, so > > >> if that isn't a sufficient filter, you could enhance that or create a > > new > > >> one backed by that class if it seems appropriate. > > >> > > >> 3. Again, we have AsyncLogger and the AsyncAppender implementations > that > > >> fulfill this niche. Take a look at that to see how well it meets your > > goals > > >> here and if there are any needed enhancements. > > >> > > >> 4. As for host names, take a look at CassandraAppender for some useful > > >> parsing bits. There is also the FailoverAppender < > > >> https://logging.apache.org/log4j/2.x/manual/appenders.html# > > >> FailoverAppender> for a generic failover mechanism, but I don't see > the > > >> harm in including a list of hosts to use (it's how some other > appenders > > >> work due to their drivers anyways like Cassandra and Kafka). > > >> > > >> On 6 May 2017 at 13:30, Volkan Yazıcı <[email protected]> > wrote: > > >> > > >>> Hello, > > >>> > > >>> To the best of my knowledge, there is no Redis support in Log4j 2.x > > NoSQL > > >>> Appenders. (Please correct if I'm wrong.) I want to create a JIRA > > ticket > > >>> and start working on a RedisAppender. Though I do have some questions > > and > > >>> will appreciate your feedback. > > >>> > > >>> 1. I plan to use Jedis <https://github.com/xetorthio/jedis> for > > >>> client > > >>> and perform no shading/relocation. Objections? > > >>> 2. I want to introduce throttling via Guava's RateLimiter. Is that > > ok? > > >>> 3. I am thinking of a background worker Thread periodically > polling > > an > > >>> ArrayBlockingQueue with a fixed size. This will help me to > > facilitate > > >>> the > > >>> following features: > > >>> 1. Queue will act as a buffer addressing temporary > > produce-consume > > >>> rate mismatch. > > >>> 2. Background thread will create batches out of the queue, > rather > > >>> than individual inserts. > > >>> 3. Periodic polling will ensure that we avoid starvation when > not > > >>> enough messages for a batch. > > >>> 4. I will allow comma separated multiple <host>:<port> pairs to > > allow > > >>> fail-over. That is, I will create a Jedis client for each > > >>> <host>:<port> > > >>> pair and try them in given order in case of a failure. > > >>> > > >>> I'm planning to make a pitch on Monday to have some *company time* > for > > >>> this > > >>> story. So please do let me know of what you think. > > >>> > > >>> Best. > > >>> > > >> > > >> > > >> > > >> -- > > >> Matt Sicker <[email protected]> > > >> > > > > > > > > > > > > -- > > > Matt Sicker <[email protected]> > > > > > > > > > > > -- > > Matt Sicker <[email protected]> > > > -- Matt Sicker <[email protected]>
