Created the following JIRA ticket: LOG4J2-1897 <https://issues.apache.org/jira/browse/LOG4J2-1897>.
On Sat, May 6, 2017 at 9:06 PM, Volkan Yazıcı <[email protected]> wrote: > Appreciated the detailed prompt reply, thanks Matt! > > 1. Then I will go with Jedis without any Maven shade plugin hacks. > 2. BurstFilter covers rate limiting. > 3. AsyncLogger covers consumer-producer rate mismatch regulation. > 4. I've checked the CassandraAppender. It leverages the multiple > contact point feature provided by the Cassandra client. I'm thinking of > doing this manually, since none offered by Jedis. > 5. I will employ AbstractDatabaseAppender for batching. > 6. I will tackle AbstractDatabaseAppender->AbstractIpcAppender problem > later on. > > I will create a JIRA ticket right now with these goals. > > On Sat, May 6, 2017 at 8:49 PM, Matt Sicker <[email protected]> wrote: > >> Instead of forming a new thread to point this out, I'll note it here: the >> AbstractDatabaseAppender classes may be refactorable or renamable into a >> sort of AbstractIpcAppender instead as another major category of >> appenders. >> Might be worth examining, though it's something we'd help in structuring >> at >> least for other appenders and whatnot. If you have any questions on best >> practices, let us know! >> >> Also, it would be great to submit a JIRA issue for this as well so we have >> somewhere to track the change. >> >> On 6 May 2017 at 13:45, Matt Sicker <[email protected]> wrote: >> >> > Oh, I also forgot to mention that regarding batching of log messages, >> > several appenders already follow a similar pattern, so you may find some >> > reusable code there, especially if you're modeling this from the >> > NoSqlAppender/Manager abstract classes as well as the >> > AbstractDatabaseAppender/Manager classes which are beginning to sound >> > like they have a too-specific name. :) >> > >> > On 6 May 2017 at 13:43, Matt Sicker <[email protected]> wrote: >> > >> >> First of all, we love contributions, and this one sounds interesting, >> too. >> >> >> >> 1. I don't have experience with Redis (I've used other distributed >> memory >> >> caches and data grid frameworks, though), so I don't have an opinion on >> >> which library to use for it. >> >> >> >> 2. As for the RateLimiter class, I'll note that we have a BurstFilter < >> >> https://logging.apache.org/log4j/2.x/manual/filters.html#BurstFilter> >> >> which sounds similar to what you're aiming for. In general, rate >> limiting >> >> log messages would be more of a Filter concern than an Appender >> concern, so >> >> if that isn't a sufficient filter, you could enhance that or create a >> new >> >> one backed by that class if it seems appropriate. >> >> >> >> 3. Again, we have AsyncLogger and the AsyncAppender implementations >> that >> >> fulfill this niche. Take a look at that to see how well it meets your >> goals >> >> here and if there are any needed enhancements. >> >> >> >> 4. As for host names, take a look at CassandraAppender for some useful >> >> parsing bits. There is also the FailoverAppender < >> >> https://logging.apache.org/log4j/2.x/manual/appenders.html# >> >> FailoverAppender> for a generic failover mechanism, but I don't see the >> >> harm in including a list of hosts to use (it's how some other appenders >> >> work due to their drivers anyways like Cassandra and Kafka). >> >> >> >> On 6 May 2017 at 13:30, Volkan Yazıcı <[email protected]> wrote: >> >> >> >>> Hello, >> >>> >> >>> To the best of my knowledge, there is no Redis support in Log4j 2.x >> NoSQL >> >>> Appenders. (Please correct if I'm wrong.) I want to create a JIRA >> ticket >> >>> and start working on a RedisAppender. Though I do have some questions >> and >> >>> will appreciate your feedback. >> >>> >> >>> 1. I plan to use Jedis <https://github.com/xetorthio/jedis> for >> >>> client >> >>> and perform no shading/relocation. Objections? >> >>> 2. I want to introduce throttling via Guava's RateLimiter. Is that >> ok? >> >>> 3. I am thinking of a background worker Thread periodically >> polling an >> >>> ArrayBlockingQueue with a fixed size. This will help me to >> facilitate >> >>> the >> >>> following features: >> >>> 1. Queue will act as a buffer addressing temporary >> produce-consume >> >>> rate mismatch. >> >>> 2. Background thread will create batches out of the queue, >> rather >> >>> than individual inserts. >> >>> 3. Periodic polling will ensure that we avoid starvation when >> not >> >>> enough messages for a batch. >> >>> 4. I will allow comma separated multiple <host>:<port> pairs to >> allow >> >>> fail-over. That is, I will create a Jedis client for each >> >>> <host>:<port> >> >>> pair and try them in given order in case of a failure. >> >>> >> >>> I'm planning to make a pitch on Monday to have some *company time* for >> >>> this >> >>> story. So please do let me know of what you think. >> >>> >> >>> Best. >> >>> >> >> >> >> >> >> >> >> -- >> >> Matt Sicker <[email protected]> >> >> >> > >> > >> > >> > -- >> > Matt Sicker <[email protected]> >> > >> >> >> >> -- >> Matt Sicker <[email protected]> >> > >
