I brought up Snappy only because I used their off-heap API recently. Snappy is more about real time compression rather than size (I think snappy files tend to be larger than gzip files, but take less resources to compress and decompress). The idea here is to offer support via libraries using native implementations that can work with direct byte buffers, mmap'd files, or even just a file name.
With that in mind, is it so bad to offer the ability to execute an external process to compress the file? On 16 November 2017 at 09:59, Chandra <[email protected]> wrote: > > What if compression worked off-heap > off-heap compression sounds interesting. Let me check if I can find any. > > Snappy’s compression is a different altogether, I am not necessarily > looking for a different compression formats, as I’d have add support for it > in downstream. Standard bz2 , gzip would work in-fact. > > > Ideally a reliable agent something like `FileBeat` would be great for this > situation. :-/ > > Best, > Chandra > > > On 16 Nov 2017, 9:08 PM +0530, Matt Sicker <[email protected]>, wrote: > > What if compression worked off-heap like with some of the native > > implementations of codecs? I'm thinking of this one < > > https://github.com/xerial/snappy-java> for example. > > > > On 15 November 2017 at 23:41, Chandra <chandra.tungathurthi@rwth- > aachen.de > > wrote: > > > > > Guys, > > > > > > I need some input on how this handle situation: > > > > > > we are on HP/LL setting where the incoming requests are processed and > > > logged ( buffered, of course with “async logging”). There are some > > > situations where due to spikes in the volume of requests, the > compression > > > on rolling creates memory starvation. > > > > > > Now, a straight forward fix would to remove it from the “context” of > jvm > > > and use a script to monitor and compress it timely. as low-hanging the > > > solution may be, I am skeptical of this solution as the script _may_ > fail > > > leading to disk starvation(!) > > > > > > I am looking for an alternate and _full-proof_ solution for this > > > situation. Any thoughts, suggestions would be useful and appreciated. > > > > > > thanks! > > > Chandra > > > > > > On 16 Nov 2017, 10:01 AM +0530, Ralph Goers < > [email protected]>, > > > wrote: > > > > Unless someone objects I plan to start the release process for Log4j > > > 2.10 tomorrow. I had wanted to include a fix for LOG4J2-2106 but the > > > problem only occurs in rare situations and I am not sure how to fix it > yet. > > > There are a lot of other issues that deserve looking at but nothing I > can > > > see that warrants waiting on. > > > > > > > > Ralph > > > > > > > > > > > -- > > Matt Sicker <[email protected] > -- Matt Sicker <[email protected]>
