That is great thank you very much for your help > On Feb 26, 2017, at 3:36 PM, Matt Sicker <boa...@gmail.com> wrote: > > Afonso, are you on the right mailing list? > > Anyways, I've made more updates to my playground. Without implementing manual > buffering on top of RandomAccessFile or FileChannel (or the async file > channel, but that one has a bug in it right now causing OOM errors > occasionally which is probably due to a misunderstanding on how many > ByteBuffers I can queue up asynchronously before I'm supposed to stop and > wait), the Java 101 style of using new BufferedOutputStream(new > FileOutputStream(file)) is currently the fastest appender I have. I'm also > trying to implement a MappedByteBuffer version, but now I'm rediscovering the > reason why we have a ByteBufferDestination interface (plus all I'm getting as > a result are 2 GB files filled with nulls, so I'm doing something wrong here > anyways). > > I was also able to fix some strange performance issues in the tests that were > being caused by my Layout implementation, so I've swapped in a trivial one > that assumes input is in ASCII and casts chars to bytes to encode them. In > Java 9, we may be able to do something with the compacted strings thing they > added for ISO-8859-1 strings. > > There are also some additional benchmarks added that I found interesting > while comparing some approaches. For one, copying from a directly-allocated > ByteBuffer into a byte[] (via ByteBuffer::get(byte[])) is almost twice as > fast as copying from a heap-allocated ByteBuffer into a byte[]. Another > interesting thing I found was that doing new Date().toString() is faster than > Instant.now().toString(). > > On 26 February 2017 at 16:10, Afonso Murakami <murakam...@icloud.com > <mailto:murakam...@icloud.com>> wrote: > Thank you for your help and time . > >> On Feb 26, 2017, at 2:05 PM, Matt Sicker <boa...@gmail.com >> <mailto:boa...@gmail.com>> wrote: >> >> So far I've discovered that my implementation of Layout that uses >> CharsetEncoder.encode(CharBuffer, ByteBuffer, boolean) is slower than the >> version that just uses CharsetEncoder.encode(CharBuffer). That could be >> causing issues here. >> >> I did try out FileChannel, and based on my current implementations, it's >> around the same speed as using Files.newOutputStream(). My >> AsynchronousFileChannel usage must be incorrect as it's working much slower >> than the synchronous form. >> >> On 26 February 2017 at 13:03, Afonso Murakami <murakam...@icloud.com >> <mailto:murakam...@icloud.com>> wrote: >> I don’t know if that make any difference but I click on the link >> https://githhub.com/vz/nio-looger <https://githhub.com/vz/nio-looger> to see >> what is about and when I finished I sign of on my account may that screw up >> something or may be not, I’m not sure if Idid that or not, sorry. >> >>> On Feb 26, 2017, at 10:49 AM, Matt Sicker <boa...@gmail.com >>> <mailto:boa...@gmail.com>> wrote: >>> >>> I could be doing something wrong entirely here, but so far, the fastest way >>> I've found to write to a file (not including mmap yet) is via: >>> >>> new BufferedOutputStream(Files.newOutputStream(Paths.get("test.log"))) >>> >>> And this is with added synchronization on the append() method, too. Also, >>> one of my updates to the async channel version is causing OOM errors in JMH >>> now, so I broke something. >>> >>> On 26 February 2017 at 12:22, Matt Sicker <boa...@gmail.com >>> <mailto:boa...@gmail.com>> wrote: >>> I added some basic JMH tests to my repo along with a couple alternative >>> appender implementations. I got rid of the unnecessary file region locking >>> in the async file channel one, but it's still coming out quite a bit slower >>> than the RandomAccessFile and Files.newOutputStream() based appenders, >>> though that could be due to the use of Phaser (which I only added to >>> cleanly close the appender synchronously). >>> >>> On 26 February 2017 at 10:05, Matt Sicker <boa...@gmail.com >>> <mailto:boa...@gmail.com>> wrote: >>> Perhaps something got optimized by the JVM? I'll add some JMH tests to this >>> repo to try out various approaches. >>> >>> On Sat, Feb 25, 2017 at 21:12, Apache <ralph.go...@dslextreme.com >>> <mailto:ralph.go...@dslextreme.com>> wrote: >>> I tried using a FileChannel for the FileAppender a week or so ago to see if >>> passing the ByteBuffer to the FileChannel would improve performance since >>> it doesn’t have to be synchronized. I didn’t see any improvement though and >>> I ended up reverting it. But I might have done something wrong. >>> >>> Ralph >>> >>>> On Feb 25, 2017, at 4:19 PM, Matt Sicker <boa...@gmail.com >>>> <mailto:boa...@gmail.com>> wrote: >>>> >>>> We already use a bit of NIO (ByteBuffer for layouts and >>>> appenders/managers, MappedByteBuffer for mmap'd files, FileLock for >>>> locking files, etc.), and I've been playing around with the NIO API >>>> lately. I have some sample code here <https://github.com/jvz/nio-logger >>>> <https://github.com/jvz/nio-logger>> to show some trivial use case of >>>> AsynchronousFileChannel. In Java 7, there is also >>>> AsynchronousSocketChannel which could theoretically be used instead of >>>> adding Netty for a faster socket appender. In that regard, I'm curious as >>>> to how useful it would be to have similar appenders as the OutputStream >>>> ones, but instead using WritableByteChannel, GatheringByteChannel >>>> (possible parallelization of file writing?), and the async channels >>>> (there's an AsynchronousByteChannel class, but I think they screwed this >>>> one up as only one of the three async channel classes implements it). >>>> >>>> Another related issue I've seen is that in a message-oriented appender >>>> (e.g., the Kafka one), being able to stream directly to a ByteBuffer is >>>> not the right way to go about encoding log messages into the appender. >>>> Instead, I was thinking that a pool of reusable ByteBuffers could be used >>>> here where a ByteBuffer is borrowed on write and returned on completion >>>> (via a CompletionHandler callback). The Kafka client uses a similar >>>> strategy for producing messages by dynamically allocating a pool of >>>> ByteBuffers based on available memory. >>>> >>>> Also, I don't have much experience with this, but if we had a pool of >>>> reusable ByteBuffers, could we use direct allocation to get off-heap >>>> buffers? That seems like an interesting use case. >>>> >>>> -- >>>> Matt Sicker <boa...@gmail.com <mailto:boa...@gmail.com>> >>> >>> -- >>> Matt Sicker <boa...@gmail.com <mailto:boa...@gmail.com>> >>> >>> >>> >>> -- >>> Matt Sicker <boa...@gmail.com <mailto:boa...@gmail.com>> >>> >>> >>> >>> -- >>> Matt Sicker <boa...@gmail.com <mailto:boa...@gmail.com>> >> >> >> >> >> -- >> Matt Sicker <boa...@gmail.com <mailto:boa...@gmail.com>> > > > > > -- > Matt Sicker <boa...@gmail.com <mailto:boa...@gmail.com>>
signature.asc
Description: Message signed with OpenPGP using GPGMail
