I guess I am not understanding your use case quite correctly. I am thinking you have a driver that is logging and you want those logs delivered to some other location to actually be written. If that is your use case then the driver needs a log4j2.xml that configures the FlumeAppender with either the memory or file channel (depending on your needs) and points to the server(s) that is/are to receive the events. The FlumeAppender handles sending them in batches with whatever size you want (but will send them in smaller amounts if they are in the channel too long). Of course you would need the log4j-flume and flume jars. So on the driver side you wouldn’t need to write anything, just configure the appender and make sure the jars are there.
For the server that receives them you would also need Flume. Normally this would be a standalone component, but it really wouldn’t be hard to incorporate it into some other application. The only thing you would have to write would be the sink that writes the events to the database or whatever. To incorporate it into an application you would have to look at the main() method of flume and covert that to be a thread that you kick off. Ralph > On Sep 25, 2016, at 12:01 PM, Gary Gregory <garydgreg...@gmail.com> wrote: > > Hi Ralph, > > Thanks for your feedback. Flume is great in the scenarios that do not involve > sending a log buffer from the driver itself. > > I can't require a Flume Agent to be running 'on the side' for the use case > where the driver chains a log buffer at the end of the train of database IO > buffer. For completeness talking about this Flume scenario, if I read you > right, I also would need to write a custom Flume sink, which would also be in > memory, until the driver is ready to drain it. Or, I could query some other > 'safe' and 'reliable' Flume sink that the driver could then drain of events > when it needs to. > > Narrowing down on the use case where the driver chains a log buffer at the > end of the train of database IO buffer, I'll think I have to see about > converting the Log4j ListAppender into a more robust and flexible version. I > think I'll call it a CollectionAppender and allow various Collection > implementations to be plugged in. > > Gary > > Gary > > On Sat, Sep 24, 2016 at 3:44 PM, Ralph Goers <ralph.go...@dslextreme.com > <mailto:ralph.go...@dslextreme.com>> wrote: > If you are buffering events in memory you run the risk of losing events if > something should fail. > > That said, if I had your requirements I would use the FlumeAppender. It has > either an in-memory option to buffer as you are suggesting or it can write to > a local file to prevent data loss if that is a requirement. It already has > the configuration options you are looking for and has been well tested. The > only downside is that you need to have either a Flume instance receiving the > messages are something that can receive Flume events over Avro, but it is > easier just to use Flume and write a custom sink to do what you want with the > data. > > Ralph > >> On Sep 24, 2016, at 3:13 PM, Gary Gregory <garydgreg...@gmail.com >> <mailto:garydgreg...@gmail.com>> wrote: >> >> Hi All, >> >> I can't believe it, but through a convoluted use-case, I actually need an >> in-memory list appender, very much like our test-only ListAppender. >> >> The requirement is as follows. >> >> We have a JDBC driver and matching proprietary database that specializes in >> data virtualization of mainframe resources like DB2, VSAM, IMS, and all >> sorts of non-SQL data sources >> (http://www.rocketsoftware.com/products/rocket-data/rocket-data-virtualization >> >> <http://www.rocketsoftware.com/products/rocket-data/rocket-data-virtualization>) >> >> >> The high level requirement is to merge the driver log into the server's log >> for full-end to end tractability and debugging. >> >> When the driver is running on the z/OS mainframe, it can be configured with >> a z/OS specific Appender that can talk to the server log module directly. >> >> When the driver is running elsewhere, it can talk to the database via a >> Syslog socket Appender. This requires more set up on the server side and for >> the server to do special magic to know how the incoming log events match up >> with server operations. Tricky. >> >> The customer should also be able to configure the driver such that anytime >> the driver communicates to the database, it sends along whatever log events >> have accumulated since the last client-server roundtrip. This allows the >> server to match exactly the connection and operations the client performed >> with the server's own logging. >> >> In order to do that I need to buffer all log events in an Appender and when >> it's time, I need to get the list of events and reset the appender to a new >> empty list so events can keep accumulating. >> >> My proposal is to either turn our ListAppender into such an appender. For >> sanity, the appender could be configured with various sizing policies: >> >> - open: the list grows unbounded >> - closed: the list grows to a given size and _new_ events are dropped on the >> floor beyond that >> - latest: the list grows to a given size and _old_ events are dropped on the >> floor beyond that >> >> Thoughts? >> >> Gary >> >> -- >> E-Mail: garydgreg...@gmail.com <mailto:garydgreg...@gmail.com> | >> ggreg...@apache.org <mailto:ggreg...@apache.org> >> Java Persistence with Hibernate, Second Edition >> <http://www.manning.com/bauer3/> >> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> >> Spring Batch in Action <http://www.manning.com/templier/> >> Blog: http://garygregory.wordpress.com <http://garygregory.wordpress.com/> >> Home: http://garygregory.com/ <http://garygregory.com/> >> Tweet! http://twitter.com/GaryGregory <http://twitter.com/GaryGregory> > > > > -- > E-Mail: garydgreg...@gmail.com <mailto:garydgreg...@gmail.com> | > ggreg...@apache.org <mailto:ggreg...@apache.org> > Java Persistence with Hibernate, Second Edition > <http://www.manning.com/bauer3/> > JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> > Spring Batch in Action <http://www.manning.com/templier/> > Blog: http://garygregory.wordpress.com <http://garygregory.wordpress.com/> > Home: http://garygregory.com/ <http://garygregory.com/> > Tweet! http://twitter.com/GaryGregory <http://twitter.com/GaryGregory>
