I believe I see where you coming from. Please correct me if I'm wrong. Your implementation is based on the premise that a call to ReadableByteChannel.read() _initiates_ the operation and returns immediately. The OS then continues to fill the buffer while there's a free space in the buffer and the channel hasn't encountered EOF.
Is that right? > On 25 Oct 2016, at 22:16, Brunoais <brunoa...@gmail.com> wrote: > > Thank you for your time. I'll try to explain it. I hope I can clear it up. > First of it, I made a meaning mistake between asynchronous and non-blocking. > This implementation uses a non-blocking algorithm internally while providing > a blocking-like algorithm on the surface. It is single-threaded and not > multi-threaded where one thread fetches data and blocks waiting and the other > accumulates it and provides to whichever wants it. > > Second of it, I had made a mistake of going after BufferedReader instead of > going after BufferedInputStream. If you want me to go after BufferedReader > it's ok but I only thought that going after BufferedInputStream would be more > generically useful than BufferedReaderwhen I started the poc. > > On to my code: > Short answers: > • The sleep(int) exists because I don't know how to wait until more > data exists in the buffer which is part of read()'s contract. > • The ByteBuffer gives a buffer that is filled by the OS (what I > believe Channels do) instead of getting data only by demand (what I > believe Streams do). > Full answers: > The blockingFill(boolean) method is a method for a busy wait for a fill which > is used exclusively by the read() method. All other methods use the version > that does not sleep (fill(boolean)). > blockingFill(boolean)'s existance like that is only because the read() method > must not return unless either: > > • The stream ended. > • The next byte is ready for reading. > Additionally, statistically, that while loop will rarely evaluate to true as > reads are in chunks so readPos will be behind writePos most of the time. > I have no idea if an interrupt will ever happen, to be honest. The main > reasons why I'm using a sleep is because I didn't want a hog onto the CPU in > a full thread usage busy wait and because I didn't find any way of doing a > thread sleep in order to wake up later when the buffer managed by native code > has more data. > The Non-blocking part is managed by the buffer the OS keeps filling most if > not all the time. That buffer is the field > > ByteBuffer readBuffer > That's the gaining part against the plain old Buffered classes. > > > Did that make sense to you? Feel free to ask anything else you need. > > On 25/10/2016 20:52, Pavel Rappo wrote: >> I've skimmed through the code and I'm not sure I can see any asynchronicity >> (you were pointing at the lack of it in BufferedReader). >> And the mechanics of this is very puzzling to me, to be honest: >> void blockingFill(boolean forced) throws IOException { >> fill(forced); >> while (readPos == writePos) { >> try { >> Thread.sleep(100); >> } catch (InterruptedException e) { >> // An interrupt may mean more data is available >> } >> fill(forced); >> } >> } >> I thought you were suggesting that we should utilize the tools which OS >> provides >> more efficiently. Instead we have something that looks very similarly to a >> "busy loop" and... also who and when is supposed to interrupt Thread.sleep()? >> Sorry, I'm not following. Could you please explain how this is supposed to >> work? >> >>> On 24 Oct 2016, at 15:59, Brunoais <brunoa...@gmail.com> >>> wrote: >>> Attached and sending! >>> On 24/10/2016 13:48, Pavel Rappo wrote: >>> >>>> Could you please send a new email on this list with the source attached as >>>> a >>>> text file? >>>> >>>>> On 23 Oct 2016, at 19:14, Brunoais <brunoa...@gmail.com> >>>>> wrote: >>>>> Here's my poc/prototype: >>>>> >>>>> http://pastebin.com/WRpYWDJF >>>>> >>>>> I've implemented the bare minimum of the class that follows the same >>>>> contract of BufferedReader while signaling all issues I think it may have >>>>> or has in comments. >>>>> I also wrote some javadoc to help guiding through the class. >>>>> I could have used more fields from BufferedReader but the names were so >>>>> minimalistic that were confusing me. I intent to change them before >>>>> sending this to openJDK. >>>>> One of the major problems this has is long overflowing. It is major >>>>> because it is hidden, it will be extremely rare and it takes a really >>>>> long time to reproduce. There are different ways of dealing with it. From >>>>> just documenting to actually making code that works with it. >>>>> I built a simple test code for it to have some ideas about performance >>>>> and correctness. >>>>> >>>>> http://pastebin.com/eh6LFgwT >>>>> >>>>> This doesn't do a through test if it is actually working correctly but I >>>>> see no reason for it not working correctly after fixing the 2 bugs that >>>>> test found. >>>>> I'll also leave here some conclusions about speed and resource >>>>> consumption I found. >>>>> I made tests with default buffer sizes, 5000B 15_000B and 500_000B. I >>>>> noticed that, with my hardware, with the 1 530 000 000B file, I was >>>>> getting around: >>>>> In all buffers and fake work: 10~15s speed improvement ( from 90% HDD >>>>> speed to 100% HDD speed) >>>>> In all buffers and no fake work: 1~2s speed improvement ( from 90% HDD >>>>> speed to 100% HDD speed) >>>>> Changing the buffer size was giving different reading speeds but both >>>>> were quite equal in how much they would change when changing the buffer >>>>> size. >>>>> Finally, I could always confirm that I/O was always the slowest thing >>>>> while this code was running. >>>>> For the ones wondering about the file size; it is both to avoid OS cache >>>>> and to make the reading at the main use-case these objects are for (large >>>>> streams of bytes). >>>>> @Pavel, are you open for discussion now ;)? Need anything else? >>>>> On 21/10/2016 19:21, Pavel Rappo wrote: >>>>> >>>>>> Just to append to my previous email. BufferedReader wraps any Reader out >>>>>> there. >>>>>> Not specifically FileReader. While you're talking about the case of >>>>>> effective >>>>>> reading from a file. >>>>>> I guess there's one existing possibility to provide exactly what you >>>>>> need (as I >>>>>> understand it) under this method: >>>>>> /** >>>>>> * Opens a file for reading, returning a {@code BufferedReader} to read >>>>>> text >>>>>> * from the file in an efficient manner... >>>>>> ... >>>>>> */ >>>>>> java.nio.file.Files#newBufferedReader(java.nio.file.Path) >>>>>> It can return _anything_ as long as it is a BufferedReader. We can do >>>>>> it, but it >>>>>> needs to be investigated not only for your favorite OS but for other >>>>>> OSes as >>>>>> well. Feel free to prototype this and we can discuss it on the list >>>>>> later. >>>>>> Thanks, >>>>>> -Pavel >>>>>> >>>>>>> On 21 Oct 2016, at 18:56, Brunoais <brunoa...@gmail.com> >>>>>>> wrote: >>>>>>> Pavel is right. >>>>>>> In reality, I was expecting such BufferedReader to use only a single >>>>>>> buffer and have that Buffer being filled asynchronously, not in a >>>>>>> different Thread. >>>>>>> Additionally, I don't have the intention of having a larger buffer than >>>>>>> before unless stated through the API (the constructor). >>>>>>> In my idea, internally, it is supposed to use >>>>>>> java.nio.channels.AsynchronousFileChannel or equivalent. >>>>>>> It does not prevent having two buffers and I do not intent to change >>>>>>> BufferedReader itself. I'd do an BufferedAsyncReader of sorts (any name >>>>>>> suggestion is welcome as I'm an awful namer). >>>>>>> On 21/10/2016 18:38, Roger Riggs wrote: >>>>>>> >>>>>>>> Hi Pavel, >>>>>>>> I think Brunoais asking for a double buffering scheme in which the >>>>>>>> implementation of >>>>>>>> BufferReader fills (a second buffer) in parallel with the application >>>>>>>> reading from the 1st buffer >>>>>>>> and managing the swaps and async reads transparently. >>>>>>>> It would not change the API but would change the interactions between >>>>>>>> the buffered reader >>>>>>>> and the underlying stream. It would also increase memory requirements >>>>>>>> and processing >>>>>>>> by introducing or using a separate thread and the necessary >>>>>>>> synchronization. >>>>>>>> Though I think the formal interface semantics could be maintained, I >>>>>>>> have doubts >>>>>>>> about compatibility and its unintended consequences on existing >>>>>>>> subclasses, >>>>>>>> applications and libraries. >>>>>>>> $.02, Roger >>>>>>>> On 10/21/16 1:22 PM, Pavel Rappo wrote: >>>>>>>> >>>>>>>>> Off the top of my head, I would say it's not possible to change the >>>>>>>>> design of an >>>>>>>>> _extensible_ type that has been out there for 20 or so years. All >>>>>>>>> these I/O >>>>>>>>> streams from java.io were designed for simple synchronous use case. >>>>>>>>> It's not that their design is flawed in some way, it's that they >>>>>>>>> doesn't seem to >>>>>>>>> suit your needs. Have you considered using >>>>>>>>> java.nio.channels.AsynchronousFileChannel >>>>>>>>> in your applications? >>>>>>>>> -Pavel >>>>>>>>> >>>>>>>>>> On 21 Oct 2016, at 17:08, Brunoais <brunoa...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> Any feedback on this? I'm really interested in implementing such >>>>>>>>>> BufferedReader/BufferedStreamReader to allow speeding up my >>>>>>>>>> applications without having to think in an asynchronous way or >>>>>>>>>> multi-threading while programming with it. >>>>>>>>>> That's why I'm asking this here. >>>>>>>>>> On 13/10/2016 14:45, Brunoais wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> I looked at BufferedReader source code for java 9 long with the >>>>>>>>>>> source code of the channels/streams used. I noticed that, like in >>>>>>>>>>> java 7, BufferedReader does not use an Async API to load data from >>>>>>>>>>> files, instead, the data loading is all done synchronously even >>>>>>>>>>> when the OS allows requesting a file to be read and getting a >>>>>>>>>>> warning later when the file is effectively read. >>>>>>>>>>> Why Is BufferedReader not async while providing a sync API? >>>>>>>>>>> >>> <BufferedNonBlockStream.java><Tests.java> >>> >