Hi Ralph, Thanks for your feedback. Flume is great in the scenarios that do not involve sending a log buffer from the driver itself.
I can't require a Flume Agent to be running 'on the side' for the use case where the driver chains a log buffer at the end of the train of database IO buffer. For completeness talking about this Flume scenario, if I read you right, I also would need to write a custom Flume sink, which would also be in memory, until the driver is ready to drain it. Or, I could query some other 'safe' and 'reliable' Flume sink that the driver could then drain of events when it needs to. Narrowing down on the use case where the driver chains a log buffer at the end of the train of database IO buffer, I'll think I have to see about converting the Log4j ListAppender into a more robust and flexible version. I think I'll call it a CollectionAppender and allow various Collection implementations to be plugged in. Gary Gary On Sat, Sep 24, 2016 at 3:44 PM, Ralph Goers <ralph.go...@dslextreme.com> wrote: > If you are buffering events in memory you run the risk of losing events if > something should fail. > > That said, if I had your requirements I would use the FlumeAppender. It > has either an in-memory option to buffer as you are suggesting or it can > write to a local file to prevent data loss if that is a requirement. It > already has the configuration options you are looking for and has been well > tested. The only downside is that you need to have either a Flume instance > receiving the messages are something that can receive Flume events over > Avro, but it is easier just to use Flume and write a custom sink to do what > you want with the data. > > Ralph > > On Sep 24, 2016, at 3:13 PM, Gary Gregory <garydgreg...@gmail.com> wrote: > > Hi All, > > I can't believe it, but through a convoluted use-case, I actually need an > in-memory list appender, very much like our test-only ListAppender. > > The requirement is as follows. > > We have a JDBC driver and matching proprietary database that specializes > in data virtualization of mainframe resources like DB2, VSAM, IMS, and all > sorts of non-SQL data sources (http://www.rocketsoftware. > com/products/rocket-data/rocket-data-virtualization) > > The high level requirement is to merge the driver log into the server's > log for full-end to end tractability and debugging. > > When the driver is running on the z/OS mainframe, it can be configured > with a z/OS specific Appender that can talk to the server log module > directly. > > When the driver is running elsewhere, it can talk to the database via a > Syslog socket Appender. This requires more set up on the server side and > for the server to do special magic to know how the incoming log events > match up with server operations. Tricky. > > The customer should also be able to configure the driver such that anytime > the driver communicates to the database, it sends along whatever log events > have accumulated since the last client-server roundtrip. This allows the > server to match exactly the connection and operations the client performed > with the server's own logging. > > In order to do that I need to buffer all log events in an Appender and > when it's time, I need to get the list of events and reset the appender to > a new empty list so events can keep accumulating. > > My proposal is to either turn our ListAppender into such an appender. For > sanity, the appender could be configured with various sizing policies: > > - open: the list grows unbounded > - closed: the list grows to a given size and _new_ events are dropped on > the floor beyond that > - latest: the list grows to a given size and _old_ events are dropped on > the floor beyond that > > Thoughts? > > Gary > > -- > E-Mail: garydgreg...@gmail.com | ggreg...@apache.org > Java Persistence with Hibernate, Second Edition > <http://www.manning.com/bauer3/> > JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> > Spring Batch in Action <http://www.manning.com/templier/> > Blog: http://garygregory.wordpress.com > Home: http://garygregory.com/ > Tweet! http://twitter.com/GaryGregory > > > -- E-Mail: garydgreg...@gmail.com | ggreg...@apache.org Java Persistence with Hibernate, Second Edition <http://www.manning.com/bauer3/> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> Spring Batch in Action <http://www.manning.com/templier/> Blog: http://garygregory.wordpress.com Home: http://garygregory.com/ Tweet! http://twitter.com/GaryGregory
