Daniel Keep wrote:
Andrei Alexandrescu wrote:
Interesting. Could you please give more details about this? Why is
range-based I/O a bad idea, and what can we do to make it a better one?
(A clarification: I *should* have said "...basing IO entirely on ranges
is -probably- a bad idea".)
<rambling>
My concern is the interface.
Let's take a hypothetical input range that reads from a file. Since
we're parsing XML, we want it to be character data. So the interface
might look something like:
struct Stream(T)
{
T front();
bool empty();
void next();
}
(I realise I probably got at least one name wrong; I can't be bothered
digging up the exact names, and it's irrelevant anyway :P)
Yah, we had to choose popFront instead of the shorter next because there
was no obvious corresponding "txen" to extract the last element.
My concern is that front returns T: a single character.
I wrote an archival tool many, many years ago in VB. It worked by
reading and writing a single byte at a time, and naturally performed
shockingly. I knew there had to be a faster way since other programs
didn't crawl like mine was and discovered that reading/writing in larger
blocks gave significantly better performance. [1]
I see, and I'm glad to dissipate this concern. There are three
interfaces that Phobos will define: byChar, byLine, and byBlock. So you
get to choose the transfer unit and transfer mechanism. (byLine allows
you to choose the separator too.) Nowadays I use text files often so I
use byLine. It's very rare that you want to process input one character
at a time, and indeed it would suck if the infrastructure would insist
that that's the unit of transfer.
Much of the performance of Tango's IO system (and from the XML parsing
code, too) is that it operates on big arrays wherever it can. Hell, the
pull parser is, as far as anyone is able to tell, faster than every
other XML parser in existence specifically because it reads the whole
file in one IO operation and then just deals with slices and array access.
(That's great, but isn't sometimes the file a socket stream?)
I don't see this approach clashing with ranges because arrays are ranges
so this setup is very natural to implement with ranges.
That's one half of my worry with this: that the range interface
specifically precludes efficient batch operations.
Hope this went away.
Another, somewhat smaller concern, is that the range interface is
back-to-front for IO.
Consider a stream: you don't know if the stream is empty until you
attempt to read past the end of it. Standard input does this, network
sockets do this... probably others.
>
But the range interface asks "is this empty?", which you can't answer
until you attempt to read from it. So to implement .empty for a
hypothetical stdin range, you'd need to try reading past the current
location. If you get a character, you've just modified the underlying
stream.
Yah, however note that if you subsequently copy the range, the
already-read front is also copied so there's no loss. Problems appear if
you create e.g. two input ranges from the same FILE* or socket or whatnot.
Walter and I discussed this problem for a long time. I also discussed
the problem in the newsgroup. I argued that the simplest and most
natural interface for a pure input stream has only one function getNext
which at the same time gets the element and bumps the stream.
Unfortunately, since all forward ranges are also input ranges, that
interface must also work well for all other ranges (e.g. arrays), in
which case it would be contorted. We decided to define what we now have.
(Actually, this is more of a concern for me in any situation where
computing the next element of a range is an expensive operation, or an
operation with side-effects. I had the same issue when attempting to
bind coroutines to the opApply interface. You had to eagerly compute
the next value in order to answer the question: is there a next element?)
Yah but you can always cache the result of the computation. The
remaining annoyance is that the side effect occurs earlier than you'd
expect.
Maybe these won't turn out to be problems in practice. But my gut
feeling is that IO would be better served by a Tango-style interface
(putting the emphasis on efficient block transfers), with ranges
wrapping that if you're willing to maybe take a performance hit.
I think we can do better by defining a general interface that will work
for arrays as good as if hand-written.
The MSB is that ranges and block transfer are not at all in conflict.
Andrei
