Hardly. In fact, I've spent quite some time trying to figure
this out.
Doing bulk read to some array is the pushing burden of
maintaining
some buffer on the user and adding the overhead of extra copy
on
buffered streams. Not to mention that the more methods we put
in range
API the more if/else forests we produce in our algorithms.
LinearBuffer tries to address this problem in:
https://github.com/D-Programming-Language/phobos/pull/2928
This simply extends the existing std.array.Appender with
popFrontN
operations.
The use case would be like this:
1. accumulate data in the LiniearBuffer by appending to the
buffer
2. try to identify the next token
3. if a token is identified popFontN the length of the token
4. otherwise acumulate some more (go to 1.)
Yes, that's the pattern. Thoughts:
1. popFrontN is just confusing - this is not a range, let's not
pretend it is.
LinearBuffer is not meant as replacement of streams or such. It
is a low level data structure, that maybe could be used in the
implementation of the streams.
Considering the FIFO style of usage, I found popFrontN
appropriate. But it would be no problem to rename it.
2. As it stands it would require first copying into appender
which is slow. The opposite direction is usually a better match
- assume appender (or rather buffer) is filled to its capacity
with data from the file and keeping this invariant true.
This would function differently if the data comes over the
socket, usually you cannot fill the buffer.
Otherwise it looks quite similar to the sliding window
interface and that is great.
data --> window
popFrontN --> skip
put ~~> extend (probably should be just called 'more')
Sometimes (socket recv) you don't know how much extend.
Actually I've dig up my original code for buffer, amended and
integrated in one of Steven's Schveighoffer I/O package forks:
https://github.com/DmitryOlshansky/phobos/blob/new-io3/std/io/textbuf.d#L68
Now I recall it's not quite sliding window as it shrinks on
reads...
Anyhow the same pattern goes with: data, discard and extend.
I was also thinking about adding a new method:
ptrdiff_t putFrom(ptrdiff_t delegate(T[]) read);
meant to be used together with file read or socket receive
functions. In
this way the read/recv system functions can write directly
into the
input buffer without intermediate copies.
Yes, that is the missing piece in the current setup, something
got to fill the buffer directly.
Parser or any other component down the line shouldn't concern
itself with reading, just using some "more" primitive to
auto-magically extend the window by at least X bytes. That
would shrink-extend buffer and load it behind the scenes.
As I sad, LinearBuffer is meant to be used "behind the scenes".
I find Andrei's bulkRead, maybe combined with a matcher a la
boost::asio read_until, the simplest high level interface.
