Re: An IO Streams Library

[ikod via Digitalmars-d]

On Wednesday, 27 July 2016 at 06:18:07 UTC, Sönke Ludwig wrote:

Am 26.07.2016 um 16:50 schrieb Johannes Pfau:

Am Mon, 25 Jul 2016 13:10:42 +


Hello,

I don't know if it is good practice or not, but sometimes it 
make
life easier if you can put part of the data back into the 
input

stream.



Writing data back to a stream is quite uncommon. The standard 
way to

solve such problems is a peek method for buffered streams:

auto buf = stream.peek(length)
// You can now look at the data in buf
stream.read() will still return the data read by peek, no need 
to write

data back into the stream.



With the notable exception of ungetc() for C's file streams. 
But working on the byte level is something that ranges should 
be used for in D's case and not streams, because the latter 
tend to have a high call overhead. So such a feature could make 
sense for a StreamInputRange/InputStreamRange wrapper.


Yes, and here is simple use-case for this: you have to process 
input stream of data chunks mixed with separators from the 
socket. To read only data and not separators, you can either ask 
stream code to "read until separator", or you can read as much as 
possible, then split on separator, return split-tail back to 
stream and process split-head. Now all your reads from stream 
will either start from the new data chunk or continue reading 
from it.


Of course, application code can manage all this on its own, but 
why not to help application programmer keep his code clean?

Re: An IO Streams Library

[Sönke Ludwig via Digitalmars-d]

Am 26.07.2016 um 16:50 schrieb Johannes Pfau:

Am Mon, 25 Jul 2016 13:10:42 +


Hello,

I don't know if it is good practice or not, but sometimes it make
life easier if you can put part of the data back into the input
stream.



Writing data back to a stream is quite uncommon. The standard way to
solve such problems is a peek method for buffered streams:

auto buf = stream.peek(length)
// You can now look at the data in buf
stream.read() will still return the data read by peek, no need to write
data back into the stream.



With the notable exception of ungetc() for C's file streams. But working 
on the byte level is something that ranges should be used for in D's 
case and not streams, because the latter tend to have a high call 
overhead. So such a feature could make sense for a 
StreamInputRange/InputStreamRange wrapper.

Re: An IO Streams Library

[Johannes Pfau via Digitalmars-d]

Am Sun, 24 Jul 2016 13:03:01 +
schrieb Martin Nowak :

> On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:
> > I'm interested in feedback on this library. What is it missing? 
> > How can be better?  
> 
> I think making the actual read/write/open/accept/et.al. functions 
> used to talk to the kernel pluggable would be a good extension 
> point to hook in evented/threaded IO frameworks.
> This restricts the async integration to synchronous APIs, e.g. 
> suspendable Fibers or async-await, but that might be a reasonable 
> choice over nodejs-style inversion of control.
> 

Can you please elaborate why this is useful? As streams already provide
a source-independent way to process data it should always possible to
just implement another stream to wrap low level calls. So why add
another API layer? And if the API slightly varies (e.g. different open
flags) implementing another wrapper is much simpler than having one
wrapper dealing with different low level functions.

Re: An IO Streams Library

[Johannes Pfau via Digitalmars-d]

Am Mon, 25 Jul 2016 13:10:42 +
schrieb ikod :

> On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:
> > I see the subject of IO streams brought up here occasionally. 
> > The general consensus seems to be that we need something better 
> > than what Phobos provides.
> >
> > I wrote a library "io" that can work as a replacement for 
> > std.stdio, std.mmfile, std.cstream, and parts of std.stream:
> >
> > GitHub:  https://github.com/jasonwhite/io
> > Package: https://code.dlang.org/packages/io
> >
> > This library provides an input and output range interface for 
> > streams (which is more efficient if the stream is buffered). 
> > Thus, many of the wonderful range operations from std.range and 
> > std.algorithm can be used with this.
> >
> > I'm interested in feedback on this library. What is it missing? 
> > How can be better?
> >
> > I'm also interested in a discussion of what IO-related 
> > functionality people are missing in Phobos.
> >
> > Please destroy!  
> 
> Hello,
> 
> I don't know if it is good practice or not, but sometimes it make 
> life easier if you can put part of the data back into the input 
> stream.
> 

Writing data back to a stream is quite uncommon. The standard way to
solve such problems is a peek method for buffered streams:

auto buf = stream.peek(length)
// You can now look at the data in buf
stream.read() will still return the data read by peek, no need to write
data back into the stream.

Re: An IO Streams Library

[ikod via Digitalmars-d]

On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:
I see the subject of IO streams brought up here occasionally. 
The general consensus seems to be that we need something better 
than what Phobos provides.


I wrote a library "io" that can work as a replacement for 
std.stdio, std.mmfile, std.cstream, and parts of std.stream:


GitHub:  https://github.com/jasonwhite/io
Package: https://code.dlang.org/packages/io

This library provides an input and output range interface for 
streams (which is more efficient if the stream is buffered). 
Thus, many of the wonderful range operations from std.range and 
std.algorithm can be used with this.


I'm interested in feedback on this library. What is it missing? 
How can be better?


I'm also interested in a discussion of what IO-related 
functionality people are missing in Phobos.


Please destroy!


Hello,

I don't know if it is good practice or not, but sometimes it make 
life easier if you can put part of the data back into the input 
stream.

Re: An IO Streams Library

[Martin Nowak via Digitalmars-d]

On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:
I'm interested in feedback on this library. What is it missing? 
How can be better?


I think making the actual read/write/open/accept/et.al. functions 
used to talk to the kernel pluggable would be a good extension 
point to hook in evented/threaded IO frameworks.
This restricts the async integration to synchronous APIs, e.g. 
suspendable Fibers or async-await, but that might be a reasonable 
choice over nodejs-style inversion of control.

Re: An IO Streams Library

[Martin Nowak via Digitalmars-d]

On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:
I'm also interested in a discussion of what IO-related 
functionality people are missing in Phobos.


This is still a very interesting approach that could even become 
a candidate for std.io at some point. Would be great if we could 
get a few more people behind this, for sure there are already 
plenty of D socket implementations out there.


https://github.com/jasonwhite/io/pull/3#issuecomment-234775601

Re: An IO Streams Library

[Kagamin via Digitalmars-d]

On Monday, 8 February 2016 at 20:21:31 UTC, Jason White wrote:
Reads and writes are not guaranteed to fill/write the entire 
buffer you throw at it. This is what readExactly/writeExactly 
are for. Those will throw an exception if the entire read/write 
cannot be done.


You mean posix non-blocking IO?

Re: An IO Streams Library

[Chris Wright via Digitalmars-d]

On Tue, 09 Feb 2016 09:45:03 +, Kagamin wrote:

> On Monday, 8 February 2016 at 20:21:31 UTC, Jason White wrote:
>> Reads and writes are not guaranteed to fill/write the entire buffer you
>> throw at it. This is what readExactly/writeExactly are for. Those will
>> throw an exception if the entire read/write cannot be done.
> 
> You mean posix non-blocking IO?

No, Posix non-blocking IO will fill a buffer with all data that is 
currently available and tell you the number of bytes it managed to give 
you. And then it will set errno to EAGAIN.

readExactly will either fill the entire buffer or throw an exception 
saying that it was unable to do so.

It's using Posix functions underneath, so if it repeatedly calls them 
until it's sent the whole buffer (assuming EAGAIN is set), that saves me 
work sometimes (but I think phobos does that already).

But if it throws an exception even when errno is set to EAGAIN, that 
would be useful for writing to a file atomically.

Re: An IO Streams Library

[Wyatt via Digitalmars-d]

On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:


This library provides an input and output range interface for 
streams (which is more efficient if the stream is buffered). 
Thus, many of the wonderful range operations from std.range and 
std.algorithm can be used with this.


Ah, grand!  I love the idea and my impression from browsing the 
source a bit is positive enough to say I'm looking forward to 
what comes out of this.


Though I AM a little ambivalent-- I had a series of pretty 
in-depth conversations on this topic with a friend a while back 
and we came to a consensus that stream semantics are a tricky 
thing because of the historical baggage around them and how they 
tend to get conflated with other concepts.


Looking at your API design, I think you've hit close to a lot of 
the same conclusions we reached, but here are the notes I took 
for the sake of providing an additional perspective:
http://radiusic.com/doc/streamNotes (Sorry, I tried just pasting 
them and it was moderately unreadable even in the preview)


I think the most important things we hit upon are:
1. A stream is fundamentally unidirectional.
2. A stream is raw, untyped data that becomes a range through an 
adapter that mediates access.


-Wyatt

Re: An IO Streams Library

[Dejan Lekic via Digitalmars-d]

I think this is a huge task and requires a (huge) DIP, and 
collaborative effort of coming up with a good, really good, API 
for BOTH synchronous and asynchronous IO. As mentioned in the 
previous messages, there is already an asyncio library, although 
I am not sure it is good enough to be in Phobos...


I know people may not like Boost, but Boost.Asio is amazing and 
we should perhaps use it as a guide, if not copy the whole API...

Re: An IO Streams Library

[Ola Fosheim Grøstad via Digitalmars-d]

On Monday, 8 February 2016 at 09:12:25 UTC, Jakob Ovrum wrote:
I don't think we've used DIPs for library additions before. I 
don't see what it would provide over module documentation.


Probably because a standard library should keep things simple, 
meaning it should unify high level  functionality on aggregates 
as ranges/iterators or a generalization over streams/ranges that 
is as simple as possible across the board.


If not, then why have it in the standard library?

Re: An IO Streams Library

[Jakob Ovrum via Digitalmars-d]

On Monday, 8 February 2016 at 09:08:53 UTC, Dejan Lekic wrote:
I think this is a huge task and requires a (huge) DIP, and 
collaborative effort of coming up with a good, really good, API 
for BOTH synchronous and asynchronous IO. As mentioned in the 
previous messages, there is already an asyncio library, 
although I am not sure it is good enough to be in Phobos...


I don't think we've used DIPs for library additions before. I 
don't see what it would provide over module documentation.

Re: An IO Streams Library

[Atila Neves via Digitalmars-d]

On Monday, 8 February 2016 at 09:08:53 UTC, Dejan Lekic wrote:
I think this is a huge task and requires a (huge) DIP, and 
collaborative effort of coming up with a good, really good, API 
for BOTH synchronous and asynchronous IO. As mentioned in the 
previous messages, there is already an asyncio library, 
although I am not sure it is good enough to be in Phobos...


I know people may not like Boost, but Boost.Asio is amazing and 
we should perhaps use it as a guide, if not copy the whole 
API...


I like boost. Well, sometimes. I _severely_ dislike boost::asio. 
The new couroutines may make it easier to use, I'd have to try 
1st, but what's the point when vibe.d exists?


Atila

Re: An IO Streams Library

[Kagamin via Digitalmars-d]

On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:

GitHub:  https://github.com/jasonwhite/io


In the output stream you compare output data length to the input 
data length. In case of a transcoding stream they can be 
different. Are you trying to account for partial writes?

Re: An IO Streams Library

[Atila Neves via Digitalmars-d]

On Monday, 8 February 2016 at 17:11:56 UTC, Chris Wright wrote:

On Mon, 08 Feb 2016 12:19:59 +, Atila Neves wrote:


On Monday, 8 February 2016 at 09:08:53 UTC, Dejan Lekic wrote:

[...]


I like boost. Well, sometimes. I _severely_ dislike 
boost::asio.
The new couroutines may make it easier to use, I'd have to try 
1st, but

what's the point when vibe.d exists?

Atila


I can't use vibe.d for one of my projects because I need to 
keep tight control of coroutine scheduling. Better async IO 
routines in Phobos would help me -- though I'm mainly looking 
for higher level stuff, like a telnet handler.


You can always use fibers yourself and just do the IO there. It's 
cooperative multithreading so you control when one of them yields.


Atila

Re: An IO Streams Library

[Chris Wright via Digitalmars-d]

On Mon, 08 Feb 2016 12:19:59 +, Atila Neves wrote:

> On Monday, 8 February 2016 at 09:08:53 UTC, Dejan Lekic wrote:
>> I think this is a huge task and requires a (huge) DIP, and
>> collaborative effort of coming up with a good, really good, API for
>> BOTH synchronous and asynchronous IO. As mentioned in the previous
>> messages, there is already an asyncio library, although I am not sure
>> it is good enough to be in Phobos...
>>
>> I know people may not like Boost, but Boost.Asio is amazing and we
>> should perhaps use it as a guide, if not copy the whole API...
> 
> I like boost. Well, sometimes. I _severely_ dislike boost::asio.
> The new couroutines may make it easier to use, I'd have to try 1st, but
> what's the point when vibe.d exists?
> 
> Atila

I can't use vibe.d for one of my projects because I need to keep tight 
control of coroutine scheduling. Better async IO routines in Phobos would 
help me -- though I'm mainly looking for higher level stuff, like a 
telnet handler.

Re: An IO Streams Library

[Jason White via Digitalmars-d]

On Monday, 8 February 2016 at 07:50:33 UTC, Jakob Ovrum wrote:
I like what I've seen so far, but I'd just like to note that 
it's easier to give feedback on the API when there is web 
documentation. GitHub Pages would be a natural place to host it.


A lot of D libraries on GitHub do this and not everyone uses 
the same tools, but for one example, here's LuaD[1] with 
reference documentation on GitHub pages, automatically 
generated and pushed by Travis-CI for the master branch.


https://github.com/JakobOvrum/LuaD


Done: https://jasonwhite.github.io/io/

(Nice work on bootDoc!)

I haven't tied it in with the Travis build yet, but that 
shouldn't be too hard to do.


Pretty docs typically helps motivate me to write more.

Re: An IO Streams Library

[Jason White via Digitalmars-d]

On Monday, 8 February 2016 at 12:02:08 UTC, Kagamin wrote:
In the output stream you compare output data length to the 
input data length. In case of a transcoding stream they can be 
different. Are you trying to account for partial writes?


Reads and writes are not guaranteed to fill/write the entire 
buffer you throw at it. This is what readExactly/writeExactly are 
for. Those will throw an exception if the entire read/write 
cannot be done.

Re: An IO Streams Library

[Chris Wright via Digitalmars-d]

On Mon, 08 Feb 2016 18:52:52 +, Atila Neves wrote:

> On Monday, 8 February 2016 at 17:11:56 UTC, Chris Wright wrote:
>> On Mon, 08 Feb 2016 12:19:59 +, Atila Neves wrote:
>>
>>> On Monday, 8 February 2016 at 09:08:53 UTC, Dejan Lekic wrote:
 [...]
>>> 
>>> I like boost. Well, sometimes. I _severely_ dislike boost::asio.
>>> The new couroutines may make it easier to use, I'd have to try 1st,
>>> but what's the point when vibe.d exists?
>>> 
>>> Atila
>>
>> I can't use vibe.d for one of my projects because I need to keep tight
>> control of coroutine scheduling. Better async IO routines in Phobos
>> would help me -- though I'm mainly looking for higher level stuff, like
>> a telnet handler.
> 
> You can always use fibers yourself and just do the IO there. It's
> cooperative multithreading so you control when one of them yields.

Which is exactly what I'm doing now.

The point is, it's useful to have multiple types of concurrency handling 
with IO, and async IO that's likely to end up in Phobos doesn't 
necessarily obviate vibe.d or duplicate work.

Re: An IO Streams Library

[Jakob Ovrum via Digitalmars-d]

On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:
I'm interested in feedback on this library. What is it missing? 
How can be better?


I like what I've seen so far, but I'd just like to note that it's 
easier to give feedback on the API when there is web 
documentation. GitHub Pages would be a natural place to host it.


A lot of D libraries on GitHub do this and not everyone uses the 
same tools, but for one example, here's LuaD[1] with reference 
documentation on GitHub pages, automatically generated and pushed 
by Travis-CI for the master branch.


https://github.com/JakobOvrum/LuaD

Re: An IO Streams Library

[Johannes Pfau via Digitalmars-d]

Am Sun, 07 Feb 2016 00:48:54 +
schrieb Jason White <54f9byee3...@gmail.com>:

> I see the subject of IO streams brought up here occasionally. The 
> general consensus seems to be that we need something better than 
> what Phobos provides.
> 
> I wrote a library "io" that can work as a replacement for 
> std.stdio, std.mmfile, std.cstream, and parts of std.stream:
> 
>  GitHub:  https://github.com/jasonwhite/io
>  Package: https://code.dlang.org/packages/io
> 
> This library provides an input and output range interface for 
> streams (which is more efficient if the stream is buffered). 
> Thus, many of the wonderful range operations from std.range and 
> std.algorithm can be used with this.
> 
> I'm interested in feedback on this library. What is it missing? 
> How can be better?
> 
> I'm also interested in a discussion of what IO-related 
> functionality people are missing in Phobos.
> 
> Please destroy!

I saw this on code.dlang.org some time ago and had a quick look. First
of all this would have to go into phobos to make sure it's used as some
kind of a standard. Conflicting stream libraries would only cause more
trouble.

Then if you want to go for phobos inclusion I'd recommend looking at
other stream implementations and learning from their mistakes ;-)
There's
https://github.com/schveiguy/phobos/tree/babe9fe338f03cafc0fb50fc0d37ea96505da3e3/std/io
which was supposed to be a stream replacement for phobos. Then there
are also vibe.d streams*.


Your Stream interfaces looks like standard stream implementations (which
is a good thing) which also work for unbuffered streams. I think it's a
good idea to support partial reads and writes. For an explanation why
partial reads, see the vibe.d rant below. Partial writes are useful
as a write syscall can be interrupted by posix signals to stop the
write. I'm not sure if the API should expose this feature (e.g. by
returning a partial write on EINTR) but it can sometimes be useful.
Still readExactly / writeAll helpers functions are useful. I would try
to implement these as UFCS functions instead of as a struct wrapper.

For some streams you'll need a TimeoutException. An interesting
question is whether users should be able to recover from
TimeoutExceptions. This essentially means if a read/write function
internally calls read/write posix calls more than once and only the
last one timed out, we already processed some data and it's not
possible to recover from a TimeoutException if the amount of already
processed data is unknown.
The simplest solution is using only one syscall internally. Then
TimeoutException => no data was processed. But this doesn't work for
read/writeExcatly (Another reason why read/writeExactly shouldn't be
the default. vibe.d...)

Regarding buffers / sliding windows I'd have a look at
https://github.com/schveiguy/phobos/blob/babe9fe338f03cafc0fb50fc0d37ea96505da3e3/std/io/buffer.d

Another design question is whether there should be an interface for
such buffered streams or whether it's OK to have only unbuffered
streams + one buffer struct / class. Basically the question is whether
there might be streams that can offer a buffer interface but can't  use
the standard implementation.




* vibe.d stream rant ahead:

vibe.d streams get some things right and some things very wrong. For
example their leastSize/empty/read combo means you might actually
have to implement reading data in any of these functions. Users have to
handle timeouts or other errors for any of these as well.

Then the API requires a buffered stream, it simply won't work for
unbuffered IO (leastSize, empty). And the fact that read reads exactly
n bytes makes stream implementations more complicated (re-reading until
enough data has been read should be done by a generic function, not
reimplemented in every stream). It even makes some user code more
complicated: I've implemented a serial port library for vibe-d.
If I don't know how many bytes will arrive with the next packet, the
read posix function usually returns the expected/available amount of
data. But now vibe.d requires me to specify a fixed length when calling
the stream read method. This leads to ugly code using peak...

Then vibe.d also mixes the sliding window / buffer concept into the
stream class, but does so in a bad way. A sliding window should expose
the internal buffer so that it's possible to consume bytes from the
buffer, skip bytes, refill... In vibe.d you can peak at the buffer. But
you can't discard data. You'll have to call read instead which copies
from the internal buffer to an external buffer, even if you only want
to skip data. Even worse, your external buffer size is limited. So you
have to implement some loop logic if you want to skip more data than
fits your buffer. And all you need is a discard(size_t n) function which
does _buffer = _buffer[n .. $] in the stream class...

TLDR: API design is very important.

Re: An IO Streams Library

[Jason White via Digitalmars-d]

On Sunday, 7 February 2016 at 10:50:24 UTC, Johannes Pfau wrote:
I saw this on code.dlang.org some time ago and had a quick 
look. First of all this would have to go into phobos to make 
sure it's used as some kind of a standard. Conflicting stream 
libraries would only cause more trouble.


Then if you want to go for phobos inclusion I'd recommend 
looking at
other stream implementations and learning from their mistakes 
;-)

There's
https://github.com/schveiguy/phobos/tree/babe9fe338f03cafc0fb50fc0d37ea96505da3e3/std/io
which was supposed to be a stream replacement for phobos. Then 
there

are also vibe.d streams*.


I saw Steven's stream implementation quite some time ago and I 
had a look at vibe's stream implementation just now. I think it 
is a mistake to use classes over structs for this sort of thing. 
I briefly tried implementing it with classes, but ran into 
problems. The non-deterministic destruction of classes is 
probably the biggest issue. One has to be careful about calling 
f.close() in order to avoid accumulating too many open file 
descriptors in programs that open a lot of files. Reference 
counting takes care of this problem nicely and has less overhead. 
This is one area where classes relying on the GC is not ideal. 
Rust's ownership system solves this problem quite well. Python 
also solves this with "with" statements.


Your Stream interfaces looks like standard stream 
implementations (which
is a good thing) which also work for unbuffered streams. I 
think it's a
good idea to support partial reads and writes. For an 
explanation why
partial reads, see the vibe.d rant below. Partial writes are 
useful
as a write syscall can be interrupted by posix signals to stop 
the
write. I'm not sure if the API should expose this feature (e.g. 
by
returning a partial write on EINTR) but it can sometimes be 
useful.


I don't want to assume what the user wants to do in the event of 
an EINTR unless a certain behavior is desired 100% of the time. I 
don't think that is the case here. Thus, that is probably 
something the user should handle manually, if needed.


Still readExactly / writeAll helpers functions are useful. I 
would try
to implement these as UFCS functions instead of as a struct 
wrapper.


I agree. I went ahead and made that change.


For some streams you'll need a TimeoutException. An interesting
question is whether users should be able to recover from
TimeoutExceptions. This essentially means if a read/write 
function
internally calls read/write posix calls more than once and only 
the

last one timed out, we already processed some data and it's not
possible to recover from a TimeoutException if the amount of 
already

processed data is unknown.
The simplest solution is using only one syscall internally. Then
TimeoutException => no data was processed. But this doesn't 
work for
read/writeExcatly (Another reason why read/writeExactly 
shouldn't be

the default. vibe.d...)


In the current implementation of readExactly/writeExactly, one 
cannot assume how much was read or written in the event of an 
exception anyway. The only way around this I can see is to return 
the number of bytes read/written in the exception itself. In 
fact, that might solve the TimeoutException problem, too. Hmm...


I'd like to keep the fundamental read/write functions at just one 
system call each in order to guarantee that they are atomic in 
relation to each other.


Regarding buffers / sliding windows I'd have a look at 
https://github.com/schveiguy/phobos/blob/babe9fe338f03cafc0fb50fc0d37ea96505da3e3/std/io/buffer.d


Another design question is whether there should be an interface 
for such buffered streams or whether it's OK to have only 
unbuffered streams + one buffer struct / class. Basically the 
question is whether there might be streams that can offer a 
buffer interface but can't  use the standard implementation.


I think it's OK to re-implement buffering for different types of 
streams where it is more efficient to do so. For example, there 
is no need to implement buffering for an in-memory stream 
because, by definition, it is already buffered.


I'm not sure if having multiple buffering strategies would be 
useful. Right now, there is only the fixed-sized sliding window. 
If multiple buffering strategies are useful, then it makes sense 
to have all streams unbuffered by default and have separate 
buffering implementations.


There is an interesting buffering approach here that is mainly 
geared towards parsing: 
https://github.com/DmitryOlshansky/datapicked/blob/master/dpick/buffer/buffer.d



* vibe.d stream rant ahead:

vibe.d streams get some things right and some things very 
wrong. For
example their leastSize/empty/read combo means you might 
actually
have to implement reading data in any of these functions. Users 
have to

handle timeouts or other errors for any of these as well.

Then the API requires a buffered stream, it simply won't work 
for
unbuffered IO (leastSize, empty). 

Re: An IO Streams Library

[cym13 via Digitalmars-d]

On Sunday, 7 February 2016 at 00:48:54 UTC, Jason White wrote:
I see the subject of IO streams brought up here occasionally. 
The general consensus seems to be that we need something better 
than what Phobos provides.

[...]


From what I can see without testing it, very nice work, thanks!

More a little surprise than anything serious though, why did you 
choose to go with "println" instead of "writeln" and such? I find 
it more confusing than anything given phobos choice.

Re: An IO Streams Library

[Jason White via Digitalmars-d]

On Sunday, 7 February 2016 at 01:59:43 UTC, Rikki Cattermole 
wrote:

Actually I think there are plenty of use cases not implemented.
Done properly as a full replacement and rework of Phobos will 
mean you need to do almost everything in e.g. std.stdio and 
std.socket but with better abstractions.


I think I'll tackle implementing sockets next. I might need that 
for another project of mine.


Once this gets polished enough, it would be great to eventually 
replace those modules in Phobos. However, it would be difficult 
to do this without compatibility breakages. For example, since 
std.stdio.File uses FILE* under the covers and this uses plain 
old file descriptors, programs that rely on that behavior would 
break.


Of course your goal may not be inline with my assertions of 
reworking Phobos. So feel free to ignore, it just would be a 
shame since it really needs some love.


My primary goal is to provide a more useful and powerful IO 
library than what Phobos provides since that is what I need for 
my other projects. That goal is not necessarily counter to 
reworking Phobos. ;)

Re: An IO Streams Library

[Rikki Cattermole via Digitalmars-d]

On 07/02/16 3:43 PM, Jason White wrote:

On Sunday, 7 February 2016 at 01:59:43 UTC, Rikki Cattermole wrote:

Actually I think there are plenty of use cases not implemented.
Done properly as a full replacement and rework of Phobos will mean you
need to do almost everything in e.g. std.stdio and std.socket but with
better abstractions.


I think I'll tackle implementing sockets next. I might need that for
another project of mine.


I wouldn't actually implement it based upon std.socket.
Use libasync instead.
https://github.com/etcimon/libasync

There has been talk about getting that into Phobos but it still needs 
time to mature.


One other important thing to note about sockets.
For anything performance related you need to have a central way to 
implement an event loop.


The one I've implemented is designed to work as a replacement for 
libasync's and to be used in Phobos.

https://github.com/rikkimax/alphaPhobos/blob/master/source/std/experimental/platform.d

If you do intend to make it compatible, you can ignore all of the 
windowing and related methods. Just keep things like optimizedEventLoop, 
eventLoopIteration, setAsDefault and thePlatform, defaultPlatform all in 
there.


We can combine later on, I just want it to be compatible when it comes time.


Once this gets polished enough, it would be great to eventually replace
those modules in Phobos. However, it would be difficult to do this
without compatibility breakages. For example, since std.stdio.File uses
FILE* under the covers and this uses plain old file descriptors,
programs that rely on that behavior would break.


There is nothing wrong with breakage. Its old code. Its time to update.
But there must be a clear upgrade path.


Of course your goal may not be inline with my assertions of reworking
Phobos. So feel free to ignore, it just would be a shame since it
really needs some love.


My primary goal is to provide a more useful and powerful IO library than
what Phobos provides since that is what I need for my other projects.
That goal is not necessarily counter to reworking Phobos. ;)


Okay sweet. Also if you can, it would be great to see you on IRC.

Re: An IO Streams Library

[Rikki Cattermole via Digitalmars-d]

On 07/02/16 1:48 PM, Jason White wrote:

I see the subject of IO streams brought up here occasionally. The
general consensus seems to be that we need something better than what
Phobos provides.

I wrote a library "io" that can work as a replacement for std.stdio,
std.mmfile, std.cstream, and parts of std.stream:

 GitHub:  https://github.com/jasonwhite/io
 Package: https://code.dlang.org/packages/io

This library provides an input and output range interface for streams
(which is more efficient if the stream is buffered). Thus, many of the
wonderful range operations from std.range and std.algorithm can be used
with this.

I'm interested in feedback on this library. What is it missing? How can
be better?

I'm also interested in a discussion of what IO-related functionality
people are missing in Phobos.

Please destroy!


I posted a link to your repo a couple days ago in IRC.
Honestly? I like it. It looks reasonably well made.

There is a bit of work regarding interfaces + ranges.
I.e. Sink really should be inheriting from OutputRange!ubyte

Its no where near Phobos quality and that is ok for now.
I do think given time it could be a reasonably good base to rework 
std.socket, std.stdio, std.stream, std.cstream and std.mmfile into a 
completely new set of modules.


Most of that code it would end up replacing is I think almost 10 years 
old either way its from D1 and I think we can do better.

Re: An IO Streams Library

[Jason White via Digitalmars-d]

On Sunday, 7 February 2016 at 01:01:21 UTC, cym13 wrote:

From what I can see without testing it, very nice work, thanks!

More a little surprise than anything serious though, why did 
you choose to go with "println" instead of "writeln" and such? 
I find it more confusing than anything given phobos choice.


Thanks!

There are a couple reasons for using print/println/etc. over 
write/writeln/etc.:
 1. A module-level definition of write(Stream s, ...) would clash 
with the stream's definition of write(...).
 2. Do we mean text-serialization or byte-for-byte output when we 
say write()? With print(), it's clear that we want the arguments 
to be converted to a text representation and have that written 
that to the stream. With write(), it's clear we're writing out 
the binary representation to the stream.

Re: An IO Streams Library

[Jason White via Digitalmars-d]

On Sunday, 7 February 2016 at 01:20:26 UTC, Rikki Cattermole 
wrote:

I posted a link to your repo a couple days ago in IRC.
Honestly? I like it. It looks reasonably well made.


Thanks. I saw a link to it in a recent thread in Learn. I figured 
I'd finally make a proper post on it.



There is a bit of work regarding interfaces + ranges.
I.e. Sink really should be inheriting from OutputRange!ubyte


I haven't had much use for the interfaces, which is why they 
aren't fleshed out. Do you have any particular use cases for this 
in mind?



Its no where near Phobos quality and that is ok for now.


I agree. The documentation needs work and I imagine there are 
lots of use cases that aren't well supported. Increased 
visibility and usage definitely helps with finding the warts.


I do think given time it could be a reasonably good base to 
rework std.socket, std.stdio, std.stream, std.cstream and 
std.mmfile into a completely new set of modules.


Most of that code it would end up replacing is I think almost 
10 years old either way its from D1 and I think we can do 
better.

Re: An IO Streams Library

[Rikki Cattermole via Digitalmars-d]

On 07/02/16 2:55 PM, Jason White wrote:

On Sunday, 7 February 2016 at 01:20:26 UTC, Rikki Cattermole wrote:

I posted a link to your repo a couple days ago in IRC.
Honestly? I like it. It looks reasonably well made.


Thanks. I saw a link to it in a recent thread in Learn. I figured I'd
finally make a proper post on it.


There is a bit of work regarding interfaces + ranges.
I.e. Sink really should be inheriting from OutputRange!ubyte


I haven't had much use for the interfaces, which is why they aren't
fleshed out. Do you have any particular use cases for this in mind?


I have no use case other then range compatibility.


Its no where near Phobos quality and that is ok for now.


I agree. The documentation needs work and I imagine there are lots of
use cases that aren't well supported. Increased visibility and usage
definitely helps with finding the warts.


Actually I think there are plenty of use cases not implemented.
Done properly as a full replacement and rework of Phobos will mean you 
need to do almost everything in e.g. std.stdio and std.socket but with 
better abstractions.


Of course your goal may not be inline with my assertions of reworking 
Phobos. So feel free to ignore, it just would be a shame since it really 
needs some love.