oh... what about our own http://logging.apache.org/log4j/2.x/manual/appenders.html#MemoryMappedFileAppender
? Gary On Mon, Sep 26, 2016 at 4:59 PM, Remko Popma <remko.po...@gmail.com> wrote: > In addition to the Flume based solution, here is another alternative idea: > use Peter Lawrey's Chronicle[1] library to store log events in a memory > mapped file. > > The appender can just keep adding events without worrying about > overflowing the memory. > > The client that reads from this file can be in a separate thread (even a > separate process by the way) and can read as much as it wants, and send it > to the server. > > Serialization: You can either serialize log events to the target format > before storing them in Chronicle (so you have binary blobs in each > Chronicle excerpt), client reads these blobs and sends them to the server > as is. Or you can use the Chronicle Log4j2 appender[2] to store the events > in Chronicle format. The tests[3] show how to read LogEvent objects from > the memory mapped file, and the client would be responsible for serializing > these log events to the target format before sending data to the server. > > [1]: https://github.com/peter-lawrey/Java-Chronicle > [2]: https://github.com/OpenHFT/Chronicle-Logger > [3]: https://github.com/OpenHFT/Chronicle-Logger/blob/ > master/logger-log4j-2/src/test/java/net/openhft/chronicle/logger/log4j2/ > Log4j2IndexedChronicleTest.java > > Remko > > Sent from my iPhone > > On 2016/09/27, at 5:57, Gary Gregory <garydgreg...@gmail.com> wrote: > > Please allow me to restate the use case I have for the CollectionAppender, > which is separate from any Flume-based or Syslog-based solution, use cases > I also have. Well, I have a Syslog use case, and whether or not Flume is in > the picture will really be a larger discussion in my organization due to > the requirement to run a Flume Agent.) > > A program (like a JDBC driver already using Log4j) communicates with > another (like a DBMS, not written in Java). The client and server > communicate over a proprietary socket protocol. The client sends a list of > buffers (in one go) to the server to perform one or more operations. One > kind of buffer this protocol defines is a log buffer (where each log event > is serialized in a non-Java format.) This allows each communication from > the client to the server to say "This is what's happened up to now". What > the server does with the log buffers is not important for this discussion. > > What is important to note is that the log buffer and other buffers go to > the server in one BLOB; which is why I cannot (in this use case) send log > events by themselves anywhere. > > I see that something (a CollectionAppender) must collect log events until > the client is ready to serialize them and send them to the server. Once the > events are drained out of the Appender (in one go by just getting the > collection), events can collect in a new collection. A synchronous drain > operation would create a new collection and return the old one. > > The question becomes: What kind of temporary location can the client use > to buffer log event until drain time? A Log4j Appender is a natural place > to collect log events since the driver uses Log4j. The driver will make its > business to drain the appender and work with the events at the right time. > I am thinking that the Log4j Appender part is generic enough for inclusion > in Log4j. > > Further thoughts? > > Thank you all for reading this far! > Gary > > On Sun, Sep 25, 2016 at 1:20 PM, Ralph Goers <ralph.go...@dslextreme.com> > wrote: > >> I guess I am not understanding your use case quite correctly. I am >> thinking you have a driver that is logging and you want those logs >> delivered to some other location to actually be written. If that is your >> use case then the driver needs a log4j2.xml that configures the >> FlumeAppender with either the memory or file channel (depending on your >> needs) and points to the server(s) that is/are to receive the events. The >> FlumeAppender handles sending them in batches with whatever size you want >> (but will send them in smaller amounts if they are in the channel too >> long). Of course you would need the log4j-flume and flume jars. So on the >> driver side you wouldn’t need to write anything, just configure the >> appender and make sure the jars are there. >> >> For the server that receives them you would also need Flume. Normally >> this would be a standalone component, but it really wouldn’t be hard to >> incorporate it into some other application. The only thing you would have >> to write would be the sink that writes the events to the database or >> whatever. To incorporate it into an application you would have to look at >> the main() method of flume and covert that to be a thread that you kick off. >> >> Ralph >> >> >> >> On Sep 25, 2016, at 12:01 PM, Gary Gregory <garydgreg...@gmail.com> >> wrote: >> >> Hi Ralph, >> >> Thanks for your feedback. Flume is great in the scenarios that do not >> involve sending a log buffer from the driver itself. >> >> I can't require a Flume Agent to be running 'on the side' for the use >> case where the driver chains a log buffer at the end of the train of >> database IO buffer. For completeness talking about this Flume scenario, if >> I read you right, I also would need to write a custom Flume sink, which >> would also be in memory, until the driver is ready to drain it. Or, I could >> query some other 'safe' and 'reliable' Flume sink that the driver could >> then drain of events when it needs to. >> >> Narrowing down on the use case where the driver chains a log buffer at >> the end of the train of database IO buffer, I'll think I have to see about >> converting the Log4j ListAppender into a more robust and flexible version. >> I think I'll call it a CollectionAppender and allow various Collection >> implementations to be plugged in. >> >> Gary >> >> Gary >> >> On Sat, Sep 24, 2016 at 3:44 PM, Ralph Goers <ralph.go...@dslextreme.com> >> wrote: >> >>> If you are buffering events in memory you run the risk of losing events >>> if something should fail. >>> >>> That said, if I had your requirements I would use the FlumeAppender. It >>> has either an in-memory option to buffer as you are suggesting or it can >>> write to a local file to prevent data loss if that is a requirement. It >>> already has the configuration options you are looking for and has been well >>> tested. The only downside is that you need to have either a Flume instance >>> receiving the messages are something that can receive Flume events over >>> Avro, but it is easier just to use Flume and write a custom sink to do what >>> you want with the data. >>> >>> Ralph >>> >>> On Sep 24, 2016, at 3:13 PM, Gary Gregory <garydgreg...@gmail.com> >>> wrote: >>> >>> Hi All, >>> >>> I can't believe it, but through a convoluted use-case, I actually need >>> an in-memory list appender, very much like our test-only ListAppender. >>> >>> The requirement is as follows. >>> >>> We have a JDBC driver and matching proprietary database that specializes >>> in data virtualization of mainframe resources like DB2, VSAM, IMS, and all >>> sorts of non-SQL data sources (http://www.rocketsoftware.com >>> /products/rocket-data/rocket-data-virtualization) >>> >>> The high level requirement is to merge the driver log into the server's >>> log for full-end to end tractability and debugging. >>> >>> When the driver is running on the z/OS mainframe, it can be configured >>> with a z/OS specific Appender that can talk to the server log module >>> directly. >>> >>> When the driver is running elsewhere, it can talk to the database via a >>> Syslog socket Appender. This requires more set up on the server side and >>> for the server to do special magic to know how the incoming log events >>> match up with server operations. Tricky. >>> >>> The customer should also be able to configure the driver such that >>> anytime the driver communicates to the database, it sends along whatever >>> log events have accumulated since the last client-server roundtrip. This >>> allows the server to match exactly the connection and operations the client >>> performed with the server's own logging. >>> >>> In order to do that I need to buffer all log events in an Appender and >>> when it's time, I need to get the list of events and reset the appender to >>> a new empty list so events can keep accumulating. >>> >>> My proposal is to either turn our ListAppender into such an appender. >>> For sanity, the appender could be configured with various sizing policies: >>> >>> - open: the list grows unbounded >>> - closed: the list grows to a given size and _new_ events are dropped on >>> the floor beyond that >>> - latest: the list grows to a given size and _old_ events are dropped on >>> the floor beyond that >>> >>> Thoughts? >>> >>> Gary >>> >>> -- >>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org >>> Java Persistence with Hibernate, Second Edition >>> <http://www.manning.com/bauer3/> >>> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> >>> Spring Batch in Action <http://www.manning.com/templier/> >>> Blog: http://garygregory.wordpress.com >>> Home: http://garygregory.com/ >>> Tweet! http://twitter.com/GaryGregory >>> >>> >>> >> >> >> -- >> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org >> Java Persistence with Hibernate, Second Edition >> <http://www.manning.com/bauer3/> >> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> >> Spring Batch in Action <http://www.manning.com/templier/> >> Blog: http://garygregory.wordpress.com >> Home: http://garygregory.com/ >> Tweet! http://twitter.com/GaryGregory >> >> >> > > > -- > E-Mail: garydgreg...@gmail.com | ggreg...@apache.org > Java Persistence with Hibernate, Second Edition > <http://www.manning.com/bauer3/> > JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> > Spring Batch in Action <http://www.manning.com/templier/> > Blog: http://garygregory.wordpress.com > Home: http://garygregory.com/ > Tweet! http://twitter.com/GaryGregory > > -- E-Mail: garydgreg...@gmail.com | ggreg...@apache.org Java Persistence with Hibernate, Second Edition <http://www.manning.com/bauer3/> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> Spring Batch in Action <http://www.manning.com/templier/> Blog: http://garygregory.wordpress.com Home: http://garygregory.com/ Tweet! http://twitter.com/GaryGregory
