On Mon, Sep 26, 2016 at 6:10 PM, Gary Gregory <garydgreg...@gmail.com>
wrote:

> On Mon, Sep 26, 2016 at 6:09 PM, Gary Gregory <garydgreg...@gmail.com>
> wrote:
>
>> On Mon, Sep 26, 2016 at 5:21 PM, Ralph Goers <ralph.go...@dslextreme.com>
>> wrote:
>>
>>> I thought you didn’t want to write to a file?
>>>
>>
>> I do not but if the buffer is large enough, log events should stay in
>> RAM. But it is not quite right anyway because I'd have to interpret the
>> contents of the file to turn back into log events.
>>
>> I started reading up on the Chronicle appender; thank you Remko for
>> pointing it out.
>>
>> An appender to a cache of objects is really want I want since I also want
>> to be able to evict the cache. TBC...
>>
>
> Like a JSR-107 Appender...
>

Looking at EHCache and
https://ignite.apache.org/jcache/1.0.0/javadoc/javax/cache/Cache.html I can
see that a cache is always a kind of map, which leads to what the key
should be.

A sequence number like we have in the pattern layout seems like a natural
choice. I could see a Jsr107Appender that tracks a sequence number as the
key. The issue is that the JSR107 Cache interface defines the iterator
order as undefined which would force a client trying to drain a
Jsr107Appender to sort all entries before being able to serialize them.
Unless I can find a list-based Cache implementation within EhCache for
example.

Gary



>
> Gary
>
>>
>> Gary
>>
>>
>>> The Chronicle stuff Remko is linking to is also worth exploring.
>>>
>>> Ralph
>>>
>>>
>>>
>>> On Sep 26, 2016, at 5:04 PM, Gary Gregory <garydgreg...@gmail.com>
>>> wrote:
>>>
>>> oh... what about our own http://logging.apache.org/
>>> log4j/2.x/manual/appenders.html#MemoryMappedFileAppender
>>>
>>> ?
>>>
>>> Gary
>>>
>>> On Mon, Sep 26, 2016 at 4:59 PM, Remko Popma <remko.po...@gmail.com>
>>> wrote:
>>>
>>>> In addition to the Flume based solution, here is another alternative
>>>> idea: use Peter Lawrey's Chronicle[1] library to store log events in a
>>>> memory mapped file.
>>>>
>>>> The appender can just keep adding events without worrying about
>>>> overflowing the memory.
>>>>
>>>> The client that reads from this file can be in a separate thread (even
>>>> a separate process by the way) and can read as much as it wants, and send
>>>> it to the server.
>>>>
>>>> Serialization: You can either serialize log events to the target format
>>>> before storing them in Chronicle (so you have binary blobs in each
>>>> Chronicle excerpt), client reads these blobs and sends them to the server
>>>> as is. Or you can use the Chronicle Log4j2 appender[2] to store the events
>>>> in Chronicle format. The tests[3] show how to read LogEvent objects from
>>>> the memory mapped file, and the client would be responsible for serializing
>>>> these log events to the target format before sending data to the server.
>>>>
>>>> [1]: https://github.com/peter-lawrey/Java-Chronicle
>>>> [2]: https://github.com/OpenHFT/Chronicle-Logger
>>>> [3]: https://github.com/OpenHFT/Chronicle-Logger/blob/master
>>>> /logger-log4j-2/src/test/java/net/openhft/chronicle/logger/l
>>>> og4j2/Log4j2IndexedChronicleTest.java
>>>>
>>>> Remko
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On 2016/09/27, at 5:57, Gary Gregory <garydgreg...@gmail.com> wrote:
>>>>
>>>> Please allow me to restate the use case I have for the
>>>> CollectionAppender, which is separate from any Flume-based or Syslog-based
>>>> solution, use cases I also have. Well, I have a Syslog use case, and
>>>> whether or not Flume is in the picture will really be a larger discussion
>>>> in my organization due to the requirement to run a Flume Agent.)
>>>>
>>>> A program (like a JDBC driver already using Log4j) communicates with
>>>> another (like a DBMS, not written in Java). The client and server
>>>> communicate over a proprietary socket protocol. The client sends a list of
>>>> buffers (in one go) to the server to perform one or more operations. One
>>>> kind of buffer this protocol defines is a log buffer (where each log event
>>>> is serialized in a non-Java format.) This allows each communication from
>>>> the client to the server to say "This is what's happened up to now". What
>>>> the server does with the log buffers is not important for this discussion.
>>>>
>>>> What is important to note is that the log buffer and other buffers go
>>>> to the server in one BLOB; which is why I cannot (in this use case) send
>>>> log events by themselves anywhere.
>>>>
>>>> I see that something (a CollectionAppender) must collect log events
>>>> until the client is ready to serialize them and send them to the server.
>>>> Once the events are drained out of the Appender (in one go by just getting
>>>> the collection), events can collect in a new collection. A synchronous
>>>> drain operation would create a new collection and return the old one.
>>>>
>>>> The question becomes: What kind of temporary location can the client
>>>> use to buffer log event until drain time? A Log4j Appender is a natural
>>>> place to collect log events since the driver uses Log4j. The driver will
>>>> make its business to drain the appender and work with the events at the
>>>> right time. I am thinking that the Log4j Appender part is generic enough
>>>> for inclusion in Log4j.
>>>>
>>>> Further thoughts?
>>>>
>>>> Thank you all for reading this far!
>>>> Gary
>>>>
>>>> On Sun, Sep 25, 2016 at 1:20 PM, Ralph Goers <
>>>> ralph.go...@dslextreme.com> wrote:
>>>>
>>>>> I guess I am not understanding your use case quite correctly. I am
>>>>> thinking you have a driver that is logging and you want those logs
>>>>> delivered to some other location to actually be written.  If that is your
>>>>> use case then the driver needs a log4j2.xml that configures the
>>>>> FlumeAppender with either the memory or file channel (depending on your
>>>>> needs) and points to the server(s) that is/are to receive the events. The
>>>>> FlumeAppender handles sending them in batches with whatever size you want
>>>>> (but will send them in smaller amounts if they are in the channel too
>>>>> long). Of course you would need the log4j-flume and flume jars. So on the
>>>>> driver side you wouldn’t need to write anything, just configure the
>>>>> appender and make sure the jars are there.
>>>>>
>>>>> For the server that receives them you would also need Flume. Normally
>>>>> this would be a standalone component, but it really wouldn’t be hard to
>>>>> incorporate it into some other application. The only thing you would have
>>>>> to write would be the sink that writes the events to the database or
>>>>> whatever. To incorporate it into an application you would have to look at
>>>>> the main() method of flume and covert that to be a thread that you kick 
>>>>> off.
>>>>>
>>>>> Ralph
>>>>>
>>>>>
>>>>>
>>>>> On Sep 25, 2016, at 12:01 PM, Gary Gregory <garydgreg...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Ralph,
>>>>>
>>>>> Thanks for your feedback. Flume is great in the scenarios that do not
>>>>> involve sending a log buffer from the driver itself.
>>>>>
>>>>> I can't require a Flume Agent to be running 'on the side' for the use
>>>>> case where the driver chains a log buffer at the end of the train of
>>>>> database IO buffer. For completeness talking about this Flume scenario, if
>>>>> I read you right, I also would need to write a custom Flume sink, which
>>>>> would also be in memory, until the driver is ready to drain it. Or, I 
>>>>> could
>>>>> query some other 'safe' and 'reliable' Flume sink that the driver could
>>>>> then drain of events when it needs to.
>>>>>
>>>>> Narrowing down on the use case where the driver chains a log buffer at
>>>>> the end of the train of database IO buffer, I'll think I have to see about
>>>>> converting the Log4j ListAppender into a more robust and flexible version.
>>>>> I think I'll call it a CollectionAppender and allow various Collection
>>>>> implementations to be plugged in.
>>>>>
>>>>> Gary
>>>>>
>>>>> Gary
>>>>>
>>>>> On Sat, Sep 24, 2016 at 3:44 PM, Ralph Goers <
>>>>> ralph.go...@dslextreme.com> wrote:
>>>>>
>>>>>> If you are buffering events in memory you run the risk of losing
>>>>>> events if something should fail.
>>>>>>
>>>>>> That said, if I had your requirements I would use the FlumeAppender.
>>>>>> It has either an in-memory option to buffer as you are suggesting or it 
>>>>>> can
>>>>>> write to a local file to prevent data loss if that is a requirement. It
>>>>>> already has the configuration options you are looking for and has been 
>>>>>> well
>>>>>> tested. The only downside is that you need to have either a Flume 
>>>>>> instance
>>>>>> receiving the messages are something that can receive Flume events over
>>>>>> Avro, but it is easier just to use Flume and write a custom sink to do 
>>>>>> what
>>>>>> you want with the data.
>>>>>>
>>>>>> Ralph
>>>>>>
>>>>>> On Sep 24, 2016, at 3:13 PM, Gary Gregory <garydgreg...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I can't believe it, but through a convoluted use-case, I actually
>>>>>> need an in-memory list appender, very much like our test-only 
>>>>>> ListAppender.
>>>>>>
>>>>>> The requirement is as follows.
>>>>>>
>>>>>> We have a JDBC driver and matching proprietary database that
>>>>>> specializes in data virtualization of mainframe resources like DB2, VSAM,
>>>>>> IMS, and all sorts of non-SQL data sources (
>>>>>> http://www.rocketsoftware.com/products/rocket-data/rocket-d
>>>>>> ata-virtualization)
>>>>>>
>>>>>> The high level requirement is to merge the driver log into the
>>>>>> server's log for full-end to end tractability and debugging.
>>>>>>
>>>>>> When the driver is running on the z/OS mainframe, it can be
>>>>>> configured with a z/OS specific Appender that can talk to the server log
>>>>>> module directly.
>>>>>>
>>>>>> When the driver is running elsewhere, it can talk to the database via
>>>>>> a Syslog socket Appender. This requires more set up on the server side 
>>>>>> and
>>>>>> for the server to do special magic to know how the incoming log events
>>>>>> match up with server operations. Tricky.
>>>>>>
>>>>>> The customer should also be able to configure the driver such that
>>>>>> anytime the driver communicates to the database, it sends along whatever
>>>>>> log events have accumulated since the last client-server roundtrip. This
>>>>>> allows the server to match exactly the connection and operations the 
>>>>>> client
>>>>>> performed with the server's own logging.
>>>>>>
>>>>>> In order to do that I need to buffer all log events in an Appender
>>>>>> and when it's time, I need to get the list of events and reset the 
>>>>>> appender
>>>>>> to a new empty list so events can keep accumulating.
>>>>>>
>>>>>> My proposal is to either turn our ListAppender into such an appender.
>>>>>> For sanity, the appender could be configured with various sizing 
>>>>>> policies:
>>>>>>
>>>>>> - open: the list grows unbounded
>>>>>> - closed: the list grows to a given size and _new_ events are dropped
>>>>>> on the floor beyond that
>>>>>> - latest: the list grows to a given size and _old_ events are dropped
>>>>>> on the floor beyond that
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> Gary
>>>>>>
>>>>>> --
>>>>>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>>>>>> Java Persistence with Hibernate, Second Edition
>>>>>> <http://www.manning.com/bauer3/>
>>>>>> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
>>>>>> Spring Batch in Action <http://www.manning.com/templier/>
>>>>>> Blog: http://garygregory.wordpress.com
>>>>>> Home: http://garygregory.com/
>>>>>> Tweet! http://twitter.com/GaryGregory
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>>>>> Java Persistence with Hibernate, Second Edition
>>>>> <http://www.manning.com/bauer3/>
>>>>> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
>>>>> Spring Batch in Action <http://www.manning.com/templier/>
>>>>> Blog: http://garygregory.wordpress.com
>>>>> Home: http://garygregory.com/
>>>>> Tweet! http://twitter.com/GaryGregory
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>>>> Java Persistence with Hibernate, Second Edition
>>>> <http://www.manning.com/bauer3/>
>>>> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
>>>> Spring Batch in Action <http://www.manning.com/templier/>
>>>> Blog: http://garygregory.wordpress.com
>>>> Home: http://garygregory.com/
>>>> Tweet! http://twitter.com/GaryGregory
>>>>
>>>>
>>>
>>>
>>> --
>>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>>> Java Persistence with Hibernate, Second Edition
>>> <http://www.manning.com/bauer3/>
>>> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
>>> Spring Batch in Action <http://www.manning.com/templier/>
>>> Blog: http://garygregory.wordpress.com
>>> Home: http://garygregory.com/
>>> Tweet! http://twitter.com/GaryGregory
>>>
>>>
>>>
>>
>>
>> --
>> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
>> Java Persistence with Hibernate, Second Edition
>> <http://www.manning.com/bauer3/>
>> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
>> Spring Batch in Action <http://www.manning.com/templier/>
>> Blog: http://garygregory.wordpress.com
>> Home: http://garygregory.com/
>> Tweet! http://twitter.com/GaryGregory
>>
>
>
>
> --
> E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
> Java Persistence with Hibernate, Second Edition
> <http://www.manning.com/bauer3/>
> JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
> Spring Batch in Action <http://www.manning.com/templier/>
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
>



-- 
E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
Java Persistence with Hibernate, Second Edition
<http://www.manning.com/bauer3/>
JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
Spring Batch in Action <http://www.manning.com/templier/>
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

Reply via email to