I noticed that in MemoryMappedFileManager, if the MappedByteBuffer hasn't been fully written to when remap() is called, then the new MappedByteBuffer will include the unwritten area as well as an additional region of the file. Do you think this could cause issues with performance like we saw with the highly variable throughput in the appender? I'm guessing the idea here is to continue writing wherever we left off but while also providing a larger ByteBuffer for producers to use if the normally sized one is too small. But in that case, the producer can simply keep draining the buffer as it goes.
On 26 February 2017 at 19:27, Remko Popma <remko.po...@gmail.com> wrote: > Afonso, are you a robot? > > On Mon, Feb 27, 2017 at 8:44 AM, Afonso Murakami <murakam...@icloud.com> > wrote: > >> That is great thank you very much for your help >> >> On Feb 26, 2017, at 3:36 PM, Matt Sicker <boa...@gmail.com> wrote: >> >> Afonso, are you on the right mailing list? >> >> Anyways, I've made more updates to my playground. Without implementing >> manual buffering on top of RandomAccessFile or FileChannel (or the async >> file channel, but that one has a bug in it right now causing OOM errors >> occasionally which is probably due to a misunderstanding on how many >> ByteBuffers I can queue up asynchronously before I'm supposed to stop and >> wait), the Java 101 style of using new BufferedOutputStream(new >> FileOutputStream(file)) is currently the fastest appender I have. I'm also >> trying to implement a MappedByteBuffer version, but now I'm rediscovering >> the reason why we have a ByteBufferDestination interface (plus all I'm >> getting as a result are 2 GB files filled with nulls, so I'm doing >> something wrong here anyways). >> >> I was also able to fix some strange performance issues in the tests that >> were being caused by my Layout implementation, so I've swapped in a trivial >> one that assumes input is in ASCII and casts chars to bytes to encode them. >> In Java 9, we may be able to do something with the compacted strings thing >> they added for ISO-8859-1 strings. >> >> There are also some additional benchmarks added that I found interesting >> while comparing some approaches. For one, copying from a directly-allocated >> ByteBuffer into a byte[] (via ByteBuffer::get(byte[])) is almost twice as >> fast as copying from a heap-allocated ByteBuffer into a byte[]. Another >> interesting thing I found was that doing new Date().toString() is faster >> than Instant.now().toString(). >> >> On 26 February 2017 at 16:10, Afonso Murakami <murakam...@icloud.com> >> wrote: >> >>> Thank you for your help and time . >>> >>> On Feb 26, 2017, at 2:05 PM, Matt Sicker <boa...@gmail.com> wrote: >>> >>> So far I've discovered that my implementation of Layout that uses >>> CharsetEncoder.encode(CharBuffer, ByteBuffer, boolean) is slower than >>> the version that just uses CharsetEncoder.encode(CharBuffer). That >>> could be causing issues here. >>> >>> I did try out FileChannel, and based on my current implementations, it's >>> around the same speed as using Files.newOutputStream(). My >>> AsynchronousFileChannel usage must be incorrect as it's working much slower >>> than the synchronous form. >>> >>> On 26 February 2017 at 13:03, Afonso Murakami <murakam...@icloud.com> >>> wrote: >>> >>>> I don’t know if that make any difference but I click on the link >>>> https://githhub.com/vz/nio-looger to see what is about and when I >>>> finished I sign of on my account may that screw up something or may be not, >>>> I’m not sure if Idid that or not, sorry. >>>> >>>> On Feb 26, 2017, at 10:49 AM, Matt Sicker <boa...@gmail.com> wrote: >>>> >>>> I could be doing something wrong entirely here, but so far, the fastest >>>> way I've found to write to a file (not including mmap yet) is via: >>>> >>>> new BufferedOutputStream(Files.newOutputStream(Paths.get("test.log"))) >>>> >>>> And this is with added synchronization on the append() method, too. >>>> Also, one of my updates to the async channel version is causing OOM errors >>>> in JMH now, so I broke something. >>>> >>>> On 26 February 2017 at 12:22, Matt Sicker <boa...@gmail.com> wrote: >>>> >>>>> I added some basic JMH tests to my repo along with a couple >>>>> alternative appender implementations. I got rid of the unnecessary file >>>>> region locking in the async file channel one, but it's still coming out >>>>> quite a bit slower than the RandomAccessFile and Files.newOutputStream() >>>>> based appenders, though that could be due to the use of Phaser (which I >>>>> only added to cleanly close the appender synchronously). >>>>> >>>>> On 26 February 2017 at 10:05, Matt Sicker <boa...@gmail.com> wrote: >>>>> >>>>>> Perhaps something got optimized by the JVM? I'll add some JMH tests >>>>>> to this repo to try out various approaches. >>>>>> >>>>>> On Sat, Feb 25, 2017 at 21:12, Apache <ralph.go...@dslextreme.com> >>>>>> wrote: >>>>>> >>>>>>> I tried using a FileChannel for the FileAppender a week or so ago to >>>>>>> see if passing the ByteBuffer to the FileChannel would improve >>>>>>> performance >>>>>>> since it doesn’t have to be synchronized. I didn’t see any improvement >>>>>>> though and I ended up reverting it. But I might have done something >>>>>>> wrong. >>>>>>> >>>>>>> Ralph >>>>>>> >>>>>>> On Feb 25, 2017, at 4:19 PM, Matt Sicker <boa...@gmail.com> wrote: >>>>>>> >>>>>>> We already use a bit of NIO (ByteBuffer for layouts and >>>>>>> appenders/managers, MappedByteBuffer for mmap'd files, FileLock for >>>>>>> locking >>>>>>> files, etc.), and I've been playing around with the NIO API lately. I >>>>>>> have >>>>>>> some sample code here <https://github.com/jvz/nio-logger> to show >>>>>>> some trivial use case of AsynchronousFileChannel. In Java 7, there is >>>>>>> also >>>>>>> AsynchronousSocketChannel which could theoretically be used instead of >>>>>>> adding Netty for a faster socket appender. In that regard, I'm curious >>>>>>> as >>>>>>> to how useful it would be to have similar appenders as the OutputStream >>>>>>> ones, but instead using WritableByteChannel, GatheringByteChannel >>>>>>> (possible >>>>>>> parallelization of file writing?), and the async channels (there's an >>>>>>> AsynchronousByteChannel class, but I think they screwed this one up as >>>>>>> only >>>>>>> one of the three async channel classes implements it). >>>>>>> >>>>>>> Another related issue I've seen is that in a message-oriented >>>>>>> appender (e.g., the Kafka one), being able to stream directly to a >>>>>>> ByteBuffer is not the right way to go about encoding log messages into >>>>>>> the >>>>>>> appender. Instead, I was thinking that a pool of reusable ByteBuffers >>>>>>> could >>>>>>> be used here where a ByteBuffer is borrowed on write and returned on >>>>>>> completion (via a CompletionHandler callback). The Kafka client uses a >>>>>>> similar strategy for producing messages by dynamically allocating a >>>>>>> pool of >>>>>>> ByteBuffers based on available memory. >>>>>>> >>>>>>> Also, I don't have much experience with this, but if we had a pool >>>>>>> of reusable ByteBuffers, could we use direct allocation to get off-heap >>>>>>> buffers? That seems like an interesting use case. >>>>>>> >>>>>>> -- >>>>>>> Matt Sicker <boa...@gmail.com> >>>>>>> >>>>>>> >>>>>>> -- >>>>>> Matt Sicker <boa...@gmail.com> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Matt Sicker <boa...@gmail.com> >>>>> >>>> >>>> >>>> >>>> -- >>>> Matt Sicker <boa...@gmail.com> >>>> >>>> >>>> >>> >>> >>> -- >>> Matt Sicker <boa...@gmail.com> >>> >>> >>> >> >> >> -- >> Matt Sicker <boa...@gmail.com> >> >> >> > -- Matt Sicker <boa...@gmail.com>
