Idea is a lightweight service that starts TCP listeners that consume streams and parses them according to a layout, e.g., syslog, pipe, regex, etc. The configuration is via yaml, whereby a listener is coupled to a codec. The codec is the input stream layout coupled to the output stream log4j2 route.
Simplest use case is taking a stream, say stdout (12factor architecture), and coupling that to a log4j2 route. Other possibilities are to consume json and create log event data structures from the document. By extension, any UTF8 stream could be parsed with regex, fields of interest extracted and injecged into a log event, and passed to log4j2 to route. The log4j2 routes I have set up use a failover strategy, whereby upstream sources that go offline cause log events to be written to a local file in json format. On service recovery, I dispatch a worker that rematerializes those events and sends them upstream. Recovery begins with a rollover to a file which uses naming by convention, using the route, the worker parses the file and sends the events up. This gives me best-effort eventual consistency of log events in one or more upstream sinks. Obviously, this implies a client agent. My original architecture was based on Kafka for consistency/durability/performance but I couldn't guarantee delivery of events from the emitter to the sink. When I saw that log4j2 had failover, I came up with this solution. I just had to build the healthcheck service and recovery worker. My plan was to follow the log4j2 plugin architecture and use annotations to declare log event handlers, allowing extension to the log processor. That part's not done.
