Afonso, are you a robot? On Mon, Feb 27, 2017 at 8:44 AM, Afonso Murakami <murakam...@icloud.com> wrote:
> That is great thank you very much for your help > > On Feb 26, 2017, at 3:36 PM, Matt Sicker <boa...@gmail.com> wrote: > > Afonso, are you on the right mailing list? > > Anyways, I've made more updates to my playground. Without implementing > manual buffering on top of RandomAccessFile or FileChannel (or the async > file channel, but that one has a bug in it right now causing OOM errors > occasionally which is probably due to a misunderstanding on how many > ByteBuffers I can queue up asynchronously before I'm supposed to stop and > wait), the Java 101 style of using new BufferedOutputStream(new > FileOutputStream(file)) is currently the fastest appender I have. I'm also > trying to implement a MappedByteBuffer version, but now I'm rediscovering > the reason why we have a ByteBufferDestination interface (plus all I'm > getting as a result are 2 GB files filled with nulls, so I'm doing > something wrong here anyways). > > I was also able to fix some strange performance issues in the tests that > were being caused by my Layout implementation, so I've swapped in a trivial > one that assumes input is in ASCII and casts chars to bytes to encode them. > In Java 9, we may be able to do something with the compacted strings thing > they added for ISO-8859-1 strings. > > There are also some additional benchmarks added that I found interesting > while comparing some approaches. For one, copying from a directly-allocated > ByteBuffer into a byte[] (via ByteBuffer::get(byte[])) is almost twice as > fast as copying from a heap-allocated ByteBuffer into a byte[]. Another > interesting thing I found was that doing new Date().toString() is faster > than Instant.now().toString(). > > On 26 February 2017 at 16:10, Afonso Murakami <murakam...@icloud.com> > wrote: > >> Thank you for your help and time . >> >> On Feb 26, 2017, at 2:05 PM, Matt Sicker <boa...@gmail.com> wrote: >> >> So far I've discovered that my implementation of Layout that uses >> CharsetEncoder.encode(CharBuffer, ByteBuffer, boolean) is slower than >> the version that just uses CharsetEncoder.encode(CharBuffer). That could >> be causing issues here. >> >> I did try out FileChannel, and based on my current implementations, it's >> around the same speed as using Files.newOutputStream(). My >> AsynchronousFileChannel usage must be incorrect as it's working much slower >> than the synchronous form. >> >> On 26 February 2017 at 13:03, Afonso Murakami <murakam...@icloud.com> >> wrote: >> >>> I don’t know if that make any difference but I click on the link >>> https://githhub.com/vz/nio-looger to see what is about and when I >>> finished I sign of on my account may that screw up something or may be not, >>> I’m not sure if Idid that or not, sorry. >>> >>> On Feb 26, 2017, at 10:49 AM, Matt Sicker <boa...@gmail.com> wrote: >>> >>> I could be doing something wrong entirely here, but so far, the fastest >>> way I've found to write to a file (not including mmap yet) is via: >>> >>> new BufferedOutputStream(Files.newOutputStream(Paths.get("test.log"))) >>> >>> And this is with added synchronization on the append() method, too. >>> Also, one of my updates to the async channel version is causing OOM errors >>> in JMH now, so I broke something. >>> >>> On 26 February 2017 at 12:22, Matt Sicker <boa...@gmail.com> wrote: >>> >>>> I added some basic JMH tests to my repo along with a couple alternative >>>> appender implementations. I got rid of the unnecessary file region locking >>>> in the async file channel one, but it's still coming out quite a bit slower >>>> than the RandomAccessFile and Files.newOutputStream() based appenders, >>>> though that could be due to the use of Phaser (which I only added to >>>> cleanly close the appender synchronously). >>>> >>>> On 26 February 2017 at 10:05, Matt Sicker <boa...@gmail.com> wrote: >>>> >>>>> Perhaps something got optimized by the JVM? I'll add some JMH tests to >>>>> this repo to try out various approaches. >>>>> >>>>> On Sat, Feb 25, 2017 at 21:12, Apache <ralph.go...@dslextreme.com> >>>>> wrote: >>>>> >>>>>> I tried using a FileChannel for the FileAppender a week or so ago to >>>>>> see if passing the ByteBuffer to the FileChannel would improve >>>>>> performance >>>>>> since it doesn’t have to be synchronized. I didn’t see any improvement >>>>>> though and I ended up reverting it. But I might have done something >>>>>> wrong. >>>>>> >>>>>> Ralph >>>>>> >>>>>> On Feb 25, 2017, at 4:19 PM, Matt Sicker <boa...@gmail.com> wrote: >>>>>> >>>>>> We already use a bit of NIO (ByteBuffer for layouts and >>>>>> appenders/managers, MappedByteBuffer for mmap'd files, FileLock for >>>>>> locking >>>>>> files, etc.), and I've been playing around with the NIO API lately. I >>>>>> have >>>>>> some sample code here <https://github.com/jvz/nio-logger> to show >>>>>> some trivial use case of AsynchronousFileChannel. In Java 7, there is >>>>>> also >>>>>> AsynchronousSocketChannel which could theoretically be used instead of >>>>>> adding Netty for a faster socket appender. In that regard, I'm curious as >>>>>> to how useful it would be to have similar appenders as the OutputStream >>>>>> ones, but instead using WritableByteChannel, GatheringByteChannel >>>>>> (possible >>>>>> parallelization of file writing?), and the async channels (there's an >>>>>> AsynchronousByteChannel class, but I think they screwed this one up as >>>>>> only >>>>>> one of the three async channel classes implements it). >>>>>> >>>>>> Another related issue I've seen is that in a message-oriented >>>>>> appender (e.g., the Kafka one), being able to stream directly to a >>>>>> ByteBuffer is not the right way to go about encoding log messages into >>>>>> the >>>>>> appender. Instead, I was thinking that a pool of reusable ByteBuffers >>>>>> could >>>>>> be used here where a ByteBuffer is borrowed on write and returned on >>>>>> completion (via a CompletionHandler callback). The Kafka client uses a >>>>>> similar strategy for producing messages by dynamically allocating a pool >>>>>> of >>>>>> ByteBuffers based on available memory. >>>>>> >>>>>> Also, I don't have much experience with this, but if we had a pool of >>>>>> reusable ByteBuffers, could we use direct allocation to get off-heap >>>>>> buffers? That seems like an interesting use case. >>>>>> >>>>>> -- >>>>>> Matt Sicker <boa...@gmail.com> >>>>>> >>>>>> >>>>>> -- >>>>> Matt Sicker <boa...@gmail.com> >>>>> >>>> >>>> >>>> >>>> -- >>>> Matt Sicker <boa...@gmail.com> >>>> >>> >>> >>> >>> -- >>> Matt Sicker <boa...@gmail.com> >>> >>> >>> >> >> >> -- >> Matt Sicker <boa...@gmail.com> >> >> >> > > > -- > Matt Sicker <boa...@gmail.com> > > >
