Re: [racket-users] Why would writing to a pipe fail?

[Matthew Flatt]

At Fri, 18 Jun 2021 05:29:14 -0400, George Neuner wrote:
> My point was that the docs for write-bytes-avail et al specifically 
> mention "flush" of data, and in a way that implies (to me) that there is 
> expected to be something else underlying the buffer to "flush" to, e.g., 
> storage media, network connection, etc., ... something.  But, if I 
> understand correctly, Racket pipes are a library construct and really 
> just a named block of memory.

Agreed, but that named block of memory isn't the "buffer" in the sense
of the port API.

The intent for "buffer" in the Racket API is a thing attached to a port
where data in the buffer is not visible outside the specific port. Data
that's buffered on a file port isn't visible by reading from the file,
for example, and data that's buffered for the write end of a pipe (if
the pipe had a buffer) is not visible to the read end.

I see that I misused the word "buffer" myself in my previous response.
Where I wrote "grow the buffer", I should have written "grow the block
or memory that holds a pipe's data", or something like that.

> So it seems like "flush" is being used in the docs to mean insertion of 
> data into the buffer ...

Into the block of memory, yes. But instead of characterizing a pipe as
all buffer and no device, I'd say that pipe is unbuffered and has only
a device (for lack of a better word). All data written to a pipe is
delivered to the "device" and available to the read end of the pipe.

> which is confusing (to me) and at odds with how 
> the word typically is used in discussions of I/O.

FWIW, I really think this is the same notion of buffer as in the C
library. A `FILE*` object potentially has a buffer, `fflush` moves data
from that buffer to the underlying device, and so on. If you wrap a
Unix pipe in a C-library `FILE*`, then `fflush` pushes data from the C
library buffer (assuming the `FILE*` is made buffered) to the OS ---
even though the OS-level device is just some memory, and that memory
tends to be called as "buffer" at the OS level.

> So the limit value passed to  make-pipe  is only a maximum size for the 
> data buffer, which starts [way too] small and then grows as needed up to 
> the maximum.  Although that does make some sense - I think starting at 
> 16 bytes is a wee bit restrictive: many (most?) real world uses would 
> have to grow it very quickly.

I agree that 16 bytes is too small. Given that a port object is already
on the order of a dozen *words*, an initial size closer to 16 words
makes sense.

> Growing the buffer is expensive in any case and particularly when 
> existing data has to be copied.  With pre-allocated buffer, the I/O call 
> will spend only what time is necessary to copy new data.  That would 
> seem to be the fastest implementation possible.

There is some cost to growing a buffer, but there's also a cost to
allocating a too-large buffer, since that increases GC pressure. For
example, one existing use of pipes is in the HTTP library, where a pipe
is used to communicate incoming POST data to a decoding thread. The
pipe's limit is set to 4096 to promote streaming behavior for large
POST data, bu POST data to be much smaller than 4096 bytes in many
uses.

To get a better handle on relative costs, I tried three implementations
for pipes:

 * the current one, where the the pipe memory always grows by a factor
   of two;

 * initializing the pipe memory to 4096 bytes instead of 16 bytes; and

 * starting with 16 or (more sensibly) 64 bytes for the pipe, but grow
   the memory based on the size of the number of bytes in a write
   request instead of always doubling --- at least up to a new size of
   4096 bytes, after which the doubling algorithm takes back over.

I tried a few benchmarks:

 * a program like the one Shu-Hung posted that allocates and writes
   data to a pipe;

 * a program that alternates writes and reads within a single thread;

 * a program that writes in one thread and reads in another thread,
   with and without a limit on the pipe; and

 * a program that untgzs the Racket v8.0 source repo (because untgz
   uses a pipe to deliver unzipped data from one thread to untarring in
   another thread).

There was a measurable difference only in that first benchmark. For
writes in the range 512 to 32768 bytes, the alternative adaptive
implementations outperformed the current one by 30% on the small end of
that range and 10% for the upper end. I used 1 million iterations, and
the run time for that size range was around 100 ms. (Note that if the
data written to a pipe is later read, that would more or less cut the
difference in half.) The difference between the two new implementations
was in the noise.

Overall, although there's not much difference, the revised adaptive
implementation seems like a win for a small and easily maintained
change to the code, so I expect to push that change.

There's likely still room for improvement. Anyone who is interested
might start at "racket/src/io/port/pipe.rkt".

Re: [racket-users] Why would writing to a pipe fail?

[George Neuner]

Hi Matthew,

Sorry for the delay in replying.


On 6/17/2021 7:08 PM, Matthew Flatt wrote:

At Wed, 16 Jun 2021 17:33:56 -0400, George Neuner wrote:
> 
> Dumb question ... why should non-blocking I/O worry about "flush" at 
> all.  Why not behave like native I/O where writes are guaranteed only to 
> get to the I/O buffer?


If I understand what you mean, then that's the intent. But I think
you're using "buffer" in the sense of the OS layer. That's different
than the buffer that you have in, say, the C library. The Racket port
model uses "buffer" to mean the one like the one implemented in a C
library, and "flush" is used also used in the C library sense, and not
in terms of a buffer that might exist in the next layer (which might be
the OS or might not).


I was not necessarily thinking about any particular layer, but I was 
thinking about open files, TCP connections, etc.  I know that both [or 
neither] the OS and the I/O library may implement buffering.


My point was that the docs for write-bytes-avail et al specifically 
mention "flush" of data, and in a way that implies (to me) that there is 
expected to be something else underlying the buffer to "flush" to, e.g., 
storage media, network connection, etc., ... something.  But, if I 
understand correctly, Racket pipes are a library construct and really 
just a named block of memory.


So it seems like "flush" is being used in the docs to mean insertion of 
data into the buffer ... which is confusing (to me) and at odds with how 
the word typically is used in discussions of I/O.  To reveal prejudices, 
I've been programming for 40 years and Racket is an umpteenth language.




Probably this became more clear with experiments that Shu-Hung and you
reported, but that's what happens, except that trying to write to a
pipe whose buffer is full will then grow the buffer. Naturally, the
buffer grows by doubling --- up to the pipe's capacity limit, if any.

An OS pipe tends to have a fixed size (OS-level) buffer that is
allocated up front, and that serves as both the maximum granularity of
writes and the capacity of the pipe. Racket pipes are meant to work in
those kinds of cases, but also adapt to cases where pipes are small and
numerous or where they have no capacity limit, so the write granularity
and capacity are not so tightly coupled.


So the limit value passed to  make-pipe  is only a maximum size for the 
data buffer, which starts [way too] small and then grows as needed up to 
the maximum.  Although that does make some sense - I think starting at 
16 bytes is a wee bit restrictive: many (most?) real world uses would 
have to grow it very quickly.


I think many (most?) of us expected that Racket pipes would behave like 
OS pipes - a fixed length allocation up front - and that a single call 
to  write-bytes-avail  would fill the buffer.  [It turns out that it 
/does/ fill the buffer, but the buffer isn't what we thought it would be.]




There's likely room for improvement. Racket BC pipes absorb more in a
non-blocking write, because MzScheme was very slow in the old days, and
it was worth looping and trying harder in the low layers instead of
having to go back out to slow MzScheme code. For Racket CS, that
slow--fast boundary isn't there, and so the lower layer core tries to
stay as fast as possible by branching and looping less internally, but
I don't really know if that's the best choice.


Growing the buffer is expensive in any case and particularly when 
existing data has to be copied.  With pre-allocated buffer, the I/O call 
will spend only what time is necessary to copy new data.  That would 
seem to be the fastest implementation possible.



Anyway, thank you very much for explaining it.
George

--
You received this message because you are subscribed to the Google Groups "Racket 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/71be924c-e88e-fd91-ea94-af6694c90a0e%40comcast.net.

Re: [racket-users] Why would writing to a pipe fail?

[Matthew Flatt]

At Wed, 16 Jun 2021 17:33:56 -0400, George Neuner wrote:
> On 6/16/2021 3:45 PM, Matthew Flatt wrote:
> > At Wed, 16 Jun 2021 14:25:40 -0400, George Neuner wrote:
> > > It looks like the problem 
> > > is that "flush" is not defined ...
> >
> > Yes, "returns without blocking after writing as many bytes as it can
> > immediately flush" is vague, and more or less intentionally so. The
> > intent it really "writes as much as is convenient, with the guarantee
> > that anything written is completely flushed". Maybe the documentation
> > should be revised to say something more like that.
> >
> > There's not intended to be a guarantee that as much is written as could
> > possibly make sense by `write-bytes-avail`. Implementation issues may
> > make writing additional bytes inconvenient enough that it doesn't
> > happen, for example, even if more is always written on the next
> > `write-bytes-avail*` call. Also, ports are meant to be used in
> > concurrent settings where the notion "as much as possible" is prone to
> > race conditions.
> 
> Dumb question ... why should non-blocking I/O worry about "flush" at 
> all.  Why not behave like native I/O where writes are guaranteed only to 
> get to the I/O buffer?

If I understand what you mean, then that's the intent. But I think
you're using "buffer" in the sense of the OS layer. That's different
than the buffer that you have in, say, the C library. The Racket port
model uses "buffer" to mean the one like the one implemented in a C
library, and "flush" is used also used in the C library sense, and not
in terms of a buffer that might exist in the next layer (which might be
the OS or might not).

> > So, the intent is that you use `write-bytes` when you want to wait
> > until all bytes are written (maybe because you know that will be soon
> > enough, for whatever reason). But when using `write-bytes-avail`, be
> > prepared for a fraction of the bytes to be written, for whatever
> > reason.
> 
> It is not a problem, per se, but it is an issue that with a long string, 
> you may be forced to call  write-bytes-avail  multiple times just to be 
> filling up the port buffer.  That seems very inefficient and is at odds 
> with how native non-blocking I/O calls behave (completing or filling the 
> buffer before returning).

Probably this became more clear with experiments that Shu-Hung and you
reported, but that's what happens, except that trying to write to a
pipe whose buffer is full will then grow the buffer. Naturally, the
buffer grows by doubling --- up to the pipe's capacity limit, if any.

An OS pipe tends to have a fixed size (OS-level) buffer that is
allocated up front, and that serves as both the maximum granularity of
writes and the capacity of the pipe. Racket pipes are meant to work in
those kinds of cases, but also adapt to cases where pipes are small and
numerous or where they have no capacity limit, so the write granularity
and capacity are not so tightly coupled.

There's likely room for improvement. Racket BC pipes absorb more in a
non-blocking write, because MzScheme was very slow in the old days, and
it was worth looping and trying harder in the low layers instead of
having to go back out to slow MzScheme code. For Racket CS, that
slow--fast boundary isn't there, and so the lower layer core tries to
stay as fast as possible by branching and looping less internally, but
I don't really know if that's the best choice.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/20210617170850.17e%40sirmail.smtps.cs.utah.edu.

Re: [racket-users] Why would writing to a pipe fail?

[George Neuner]

Forgot to mention this is using  CS 8.1

--
You received this message because you are subscribed to the Google Groups "Racket 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/70ba930a-e7fe-af8a-3b57-0800d15c945c%40comcast.net.

Re: [racket-users] Why would writing to a pipe fail?

[George Neuner]

On 6/16/2021 7:01 PM, Shu-Hung You wrote:

Out of curiosity, I wrapped David's code in a loop and tried to write
509 bytes in each iteration. From the output, it looks like CS doesn't
implement pipes using a fixed-size buffer. I'm also not sure how many
different buffers there are. I think this has something to do with
George's question.
 :


Continuing in this vein, it seems that the count of bytes written starts 
small and then doubles with each successive write until finished (or 
until the buffer fills).   I'm guessing this has to do with thread 
scheduling applied to I/O ... ie. try a short write (minimum quantum), 
and if it isn't finished try a longer write (longer quantum) ???




(define msg-size 15700)
(define bstr (make-bytes msg-size 42))

(define rx-pipe-size 16384)
(define-values (rx-in rx-out) (make-pipe rx-pipe-size))

(define (available?)
  (- rx-pipe-size (pipe-content-length rx-out)))

(let loop ((start 0) (full? #f))
  (cond
    [full? (displayln "buffer full")]
    [(>= start msg-size) (displayln "done")]
    [else
 (let ((written (write-bytes-avail* bstr rx-out start)))
   (printf "written: ~a  starting at  ~a~n" written start)
   ;   (printf "available: ~a~n" (available?))
   (loop (+ start written) (= written 0))
   )
 ]))


written: 15  starting at  0
written: 16  starting at  15
written: 32  starting at  31
written: 64  starting at  63
written: 128  starting at  127
written: 256  starting at  255
written: 512  starting at  511
written: 1024  starting at  1023
written: 2048  starting at  2047
written: 4096  starting at  4095
written: 7509  starting at  8191
done


With the "msg" size changed to exceed the buffer:

written: 15  starting at  0
written: 16  starting at  15
written: 32  starting at  31
written: 64  starting at  63
written: 128  starting at  127
written: 256  starting at  255
written: 512  starting at  511
written: 1024  starting at  1023
written: 2048  starting at  2047
written: 4096  starting at  4095
written: 8192  starting at  8191
written: 1  starting at  16383
written: 0  starting at  16384
buffer full





--
You received this message because you are subscribed to the Google Groups "Racket 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/6b252cfe-ca16-073b-5164-1bb784095825%40comcast.net.

Re: [racket-users] Why would writing to a pipe fail?

[Shu-Hung You]

Out of curiosity, I wrapped David's code in a loop and tried to write
509 bytes in each iteration. From the output, it looks like CS doesn't
implement pipes using a fixed-size buffer. I'm also not sure how many
different buffers there are. I think this has something to do with
George's question.

ITERATION 0
avail: 16777216
space available? #t
pipe content length: 0
rx buffer overflow. pipe content length: 15, written 15, expected 509

ITERATION 1
avail: 16777201
space available? #t
pipe content length: 15
rx buffer overflow. pipe content length: 31, written 16, expected 509

ITERATION 2
avail: 16777185
space available? #t
pipe content length: 31
rx buffer overflow. pipe content length: 63, written 32, expected 509

ITERATION 3
avail: 16777153
space available? #t
pipe content length: 63
rx buffer overflow. pipe content length: 127, written 64, expected 509

ITERATION 4
avail: 16777089
space available? #t
pipe content length: 127
rx buffer overflow. pipe content length: 255, written 128, expected 509
done

On Wed, Jun 16, 2021 at 4:34 PM George Neuner  wrote:
>
>
> On 6/16/2021 3:45 PM, Matthew Flatt wrote:
> > At Wed, 16 Jun 2021 14:25:40 -0400, George Neuner wrote:
> > > It looks like the problem
> > > is that "flush" is not defined ...
> >
> > Yes, "returns without blocking after writing as many bytes as it can
> > immediately flush" is vague, and more or less intentionally so. The
> > intent it really "writes as much as is convenient, with the guarantee
> > that anything written is completely flushed". Maybe the documentation
> > should be revised to say something more like that.
> >
> > There's not intended to be a guarantee that as much is written as could
> > possibly make sense by `write-bytes-avail`. Implementation issues may
> > make writing additional bytes inconvenient enough that it doesn't
> > happen, for example, even if more is always written on the next
> > `write-bytes-avail*` call. Also, ports are meant to be used in
> > concurrent settings where the notion "as much as possible" is prone to
> > race conditions.
>
> Dumb question ... why should non-blocking I/O worry about "flush" at
> all.  Why not behave like native I/O where writes are guaranteed only to
> get to the I/O buffer?
>
>
> > The Racket BC and CS pipe implementations find different things
> > convenient, in this sense, and that's why they behave differently in
> > the example. (That is, it's not really about the Racket version, but CS
> > versus BC.)
> >
> > So, the intent is that you use `write-bytes` when you want to wait
> > until all bytes are written (maybe because you know that will be soon
> > enough, for whatever reason). But when using `write-bytes-avail`, be
> > prepared for a fraction of the bytes to be written, for whatever
> > reason.
>
> It is not a problem, per se, but it is an issue that with a long string,
> you may be forced to call  write-bytes-avail  multiple times just to be
> filling up the port buffer.  That seems very inefficient and is at odds
> with how native non-blocking I/O calls behave (completing or filling the
> buffer before returning).
>
> YMMV,
> George
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to racket-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/racket-users/5257581b-e863-2a2c-e3a3-802b79016281%40comcast.net.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CAMTzy%2BbU4i7Rb28HuY%3DTGh0GdHMATdJrM7pjto5jD9h%2Bf_VxOw%40mail.gmail.com.

Re: [racket-users] Why would writing to a pipe fail?

[George Neuner]

On 6/16/2021 3:45 PM, Matthew Flatt wrote:

At Wed, 16 Jun 2021 14:25:40 -0400, George Neuner wrote:
> It looks like the problem 
> is that "flush" is not defined ...


Yes, "returns without blocking after writing as many bytes as it can
immediately flush" is vague, and more or less intentionally so. The
intent it really "writes as much as is convenient, with the guarantee
that anything written is completely flushed". Maybe the documentation
should be revised to say something more like that.

There's not intended to be a guarantee that as much is written as could
possibly make sense by `write-bytes-avail`. Implementation issues may
make writing additional bytes inconvenient enough that it doesn't
happen, for example, even if more is always written on the next
`write-bytes-avail*` call. Also, ports are meant to be used in
concurrent settings where the notion "as much as possible" is prone to
race conditions.


Dumb question ... why should non-blocking I/O worry about "flush" at 
all.  Why not behave like native I/O where writes are guaranteed only to 
get to the I/O buffer?




The Racket BC and CS pipe implementations find different things
convenient, in this sense, and that's why they behave differently in
the example. (That is, it's not really about the Racket version, but CS
versus BC.)

So, the intent is that you use `write-bytes` when you want to wait
until all bytes are written (maybe because you know that will be soon
enough, for whatever reason). But when using `write-bytes-avail`, be
prepared for a fraction of the bytes to be written, for whatever
reason.


It is not a problem, per se, but it is an issue that with a long string, 
you may be forced to call  write-bytes-avail  multiple times just to be 
filling up the port buffer.  That seems very inefficient and is at odds 
with how native non-blocking I/O calls behave (completing or filling the 
buffer before returning).


YMMV,
George

--
You received this message because you are subscribed to the Google Groups "Racket 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/5257581b-e863-2a2c-e3a3-802b79016281%40comcast.net.

Re: [racket-users] Why would writing to a pipe fail?

[David Storrs]

In case anyone else runs into this issue and is worried about their code
blocking forever, here's a version that will time out.  The only change is
the use of sync/timeout and value-evt.
https://docs.racket-lang.org/value-evt/index.html

(require value-evt)
(define bstr (make-shared-bytes 509 5))
(define rx-pipe-size 16777216)
(define-values (rx-in rx-out) (make-pipe rx-pipe-size))

(define (room-in-rx-pipe? bstr)
  (define avail (- rx-pipe-size (pipe-content-length rx-out)))
  (displayln (format "avail: ~a" avail))
  (<= (bytes-length bstr) avail))

(displayln (format "space available? ~a " (room-in-rx-pipe? bstr)))
(displayln (format "pipe content length: ~a" (pipe-content-length rx-out)))

; if write-bytes takes more than 0.5 seconds then it will be interrupted
and num-bytes-written will end up being #f
; wrapping the write-bytes expressing in a thunk is important because we
don't want it to be evaluated before the sync
(define num-bytes-written (sync/timeout 0.5 (value-evt (thunk (write-bytes
bstr rx-out)
(unless (eq? num-bytes-written (bytes-length bstr))
  (displayln (format "rx buffer overflow. pipe content length: ~a, written
~a, expected ~a"
 (pipe-content-length rx-out) num-bytes-written
(bytes-length bstr

(displayln "done")

 output
avail: 16777216
space available? #t
pipe content length: 0
done
test.r

On Wed, Jun 16, 2021 at 4:13 PM David Storrs  wrote:

> Got it. Thanks.
>
> On Wed, Jun 16, 2021 at 3:45 PM Matthew Flatt  wrote:
>
>> At Wed, 16 Jun 2021 14:25:40 -0400, George Neuner wrote:
>> > It looks like the problem
>> > is that "flush" is not defined ...
>>
>> Yes, "returns without blocking after writing as many bytes as it can
>> immediately flush" is vague, and more or less intentionally so. The
>> intent it really "writes as much as is convenient, with the guarantee
>> that anything written is completely flushed". Maybe the documentation
>> should be revised to say something more like that.
>>
>> There's not intended to be a guarantee that as much is written as could
>> possibly make sense by `write-bytes-avail`. Implementation issues may
>> make writing additional bytes inconvenient enough that it doesn't
>> happen, for example, even if more is always written on the next
>> `write-bytes-avail*` call. Also, ports are meant to be used in
>> concurrent settings where the notion "as much as possible" is prone to
>> race conditions.
>>
>> The Racket BC and CS pipe implementations find different things
>> convenient, in this sense, and that's why they behave differently in
>> the example. (That is, it's not really about the Racket version, but CS
>> versus BC.)
>>
>> So, the intent is that you use `write-bytes` when you want to wait
>> until all bytes are written (maybe because you know that will be soon
>> enough, for whatever reason). But when using `write-bytes-avail`, be
>> prepared for a fraction of the bytes to be written, for whatever
>> reason.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CAE8gKocHjnCsVYcp7ubEvH_P7nX0xMzv4NmfyByeFZvSowtj2A%40mail.gmail.com.

Re: [racket-users] Why would writing to a pipe fail?

[David Storrs]

Got it. Thanks.

On Wed, Jun 16, 2021 at 3:45 PM Matthew Flatt  wrote:

> At Wed, 16 Jun 2021 14:25:40 -0400, George Neuner wrote:
> > It looks like the problem
> > is that "flush" is not defined ...
>
> Yes, "returns without blocking after writing as many bytes as it can
> immediately flush" is vague, and more or less intentionally so. The
> intent it really "writes as much as is convenient, with the guarantee
> that anything written is completely flushed". Maybe the documentation
> should be revised to say something more like that.
>
> There's not intended to be a guarantee that as much is written as could
> possibly make sense by `write-bytes-avail`. Implementation issues may
> make writing additional bytes inconvenient enough that it doesn't
> happen, for example, even if more is always written on the next
> `write-bytes-avail*` call. Also, ports are meant to be used in
> concurrent settings where the notion "as much as possible" is prone to
> race conditions.
>
> The Racket BC and CS pipe implementations find different things
> convenient, in this sense, and that's why they behave differently in
> the example. (That is, it's not really about the Racket version, but CS
> versus BC.)
>
> So, the intent is that you use `write-bytes` when you want to wait
> until all bytes are written (maybe because you know that will be soon
> enough, for whatever reason). But when using `write-bytes-avail`, be
> prepared for a fraction of the bytes to be written, for whatever
> reason.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CAE8gKofs0PPHs9mUmSgudMZsqaUEOpV6CCzRB7D3kwZyFXartg%40mail.gmail.com.

Re: [racket-users] Why would writing to a pipe fail?

[Matthew Flatt]

At Wed, 16 Jun 2021 14:25:40 -0400, George Neuner wrote:
> It looks like the problem 
> is that "flush" is not defined ...

Yes, "returns without blocking after writing as many bytes as it can
immediately flush" is vague, and more or less intentionally so. The
intent it really "writes as much as is convenient, with the guarantee
that anything written is completely flushed". Maybe the documentation
should be revised to say something more like that.

There's not intended to be a guarantee that as much is written as could
possibly make sense by `write-bytes-avail`. Implementation issues may
make writing additional bytes inconvenient enough that it doesn't
happen, for example, even if more is always written on the next
`write-bytes-avail*` call. Also, ports are meant to be used in
concurrent settings where the notion "as much as possible" is prone to
race conditions.

The Racket BC and CS pipe implementations find different things
convenient, in this sense, and that's why they behave differently in
the example. (That is, it's not really about the Racket version, but CS
versus BC.)

So, the intent is that you use `write-bytes` when you want to wait
until all bytes are written (maybe because you know that will be soon
enough, for whatever reason). But when using `write-bytes-avail`, be
prepared for a fraction of the bytes to be written, for whatever
reason.

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/20210616134541.36c%40sirmail.smtps.cs.utah.edu.

Re: [racket-users] Why would writing to a pipe fail?

[David Storrs]

For the record, it doesn't work in 8.1 either.

On Wed, Jun 16, 2021 at 2:44 PM David Storrs  wrote:

>
>
> On Wed, Jun 16, 2021 at 2:25 PM George Neuner 
> wrote:
>
>>
>> On 6/16/2021 2:16 PM, David Storrs wrote:
>>
>> Damn.  Sorry, I posted out of sync versions of code and output.  This is
>> correct:
>>
>> (define bstr (make-shared-bytes 509 5))
>> (define rx-pipe-size 16777216)
>> (define-values (rx-in rx-out) (make-pipe rx-pipe-size))
>>
>> (define (room-in-rx-pipe? bstr)
>>   (define avail (- rx-pipe-size (pipe-content-length rx-out)))
>>   (displayln (format "avail: ~a" avail))
>>   (<= (bytes-length bstr) avail))
>>
>> (displayln (format "space available? ~a " (room-in-rx-pipe? bstr)))
>> (displayln (format "pipe content length: ~a" (pipe-content-length
>> rx-out)))
>> (define num-bytes-written (write-bytes-avail* bstr rx-out))
>> (unless (eq? num-bytes-written (bytes-length bstr))
>>   (displayln (format "rx buffer overflow. pipe content length: ~a,
>> written ~a, expected ~a"
>>  (pipe-content-length rx-out) num-bytes-written
>> (bytes-length bstr
>>
>> (displayln "done")
>>
>>  output:
>>
>> avail: 16777216
>> space available? #t
>> pipe content length: 0
>> rx buffer overflow. pipe content length: 15, written 15, expected 509
>> done
>>
>>
>>
>> That's ok ... my analysis was wrong anyhow.   It looks like the problem
>> is that "flush" is not defined ... or simply doesn't work ... on Racket's
>> pipes.  *write-bytes-avail*  et al  depend on flush and so they won't
>> work.  *write-bytes*  works fine, but will block if you fill the pipe.
>>
>> This looks like something for the development team.
>>
>> George
>>
>
> Yup!  That was the problem.  It works fine in 7.6, not in 8.0.  Thanks,
> George.
>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Racket Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to racket-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/racket-users/4836be0b-f785-9394-b18f-f5228970ba44%40comcast.net
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CAE8gKodbSU2xJip-87zwfEhercsbVmjyt2AHfhcsEF%3Dq3wK0KQ%40mail.gmail.com.

Re: [racket-users] Why would writing to a pipe fail?

[David Storrs]

On Wed, Jun 16, 2021 at 2:25 PM George Neuner  wrote:

>
> On 6/16/2021 2:16 PM, David Storrs wrote:
>
> Damn.  Sorry, I posted out of sync versions of code and output.  This is
> correct:
>
> (define bstr (make-shared-bytes 509 5))
> (define rx-pipe-size 16777216)
> (define-values (rx-in rx-out) (make-pipe rx-pipe-size))
>
> (define (room-in-rx-pipe? bstr)
>   (define avail (- rx-pipe-size (pipe-content-length rx-out)))
>   (displayln (format "avail: ~a" avail))
>   (<= (bytes-length bstr) avail))
>
> (displayln (format "space available? ~a " (room-in-rx-pipe? bstr)))
> (displayln (format "pipe content length: ~a" (pipe-content-length rx-out)))
> (define num-bytes-written (write-bytes-avail* bstr rx-out))
> (unless (eq? num-bytes-written (bytes-length bstr))
>   (displayln (format "rx buffer overflow. pipe content length: ~a, written
> ~a, expected ~a"
>  (pipe-content-length rx-out) num-bytes-written
> (bytes-length bstr
>
> (displayln "done")
>
>  output:
>
> avail: 16777216
> space available? #t
> pipe content length: 0
> rx buffer overflow. pipe content length: 15, written 15, expected 509
> done
>
>
>
> That's ok ... my analysis was wrong anyhow.   It looks like the problem is
> that "flush" is not defined ... or simply doesn't work ... on Racket's
> pipes.  *write-bytes-avail*  et al  depend on flush and so they won't
> work.  *write-bytes*  works fine, but will block if you fill the pipe.
>
> This looks like something for the development team.
>
> George
>

Yup!  That was the problem.  It works fine in 7.6, not in 8.0.  Thanks,
George.

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-users/4836be0b-f785-9394-b18f-f5228970ba44%40comcast.net
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CAE8gKod26a2KC2FXyK2u9dADvQFhB90irnoGuwTi9RKXiFq34g%40mail.gmail.com.

Re: [racket-users] Why would writing to a pipe fail?

[George Neuner]

On 6/16/2021 2:16 PM, David Storrs wrote:
Damn.  Sorry, I posted out of sync versions of code and output.  This 
is correct:


(define bstr (make-shared-bytes 509 5))
(define rx-pipe-size 16777216)
(define-values (rx-in rx-out) (make-pipe rx-pipe-size))

(define (room-in-rx-pipe? bstr)
  (define avail (- rx-pipe-size (pipe-content-length rx-out)))
  (displayln (format "avail: ~a" avail))
  (<= (bytes-length bstr) avail))

(displayln (format "space available? ~a " (room-in-rx-pipe? bstr)))
(displayln (format "pipe content length: ~a" (pipe-content-length 
rx-out)))

(define num-bytes-written (write-bytes-avail* bstr rx-out))
(unless (eq? num-bytes-written (bytes-length bstr))
  (displayln (format "rx buffer overflow. pipe content length: ~a, 
written ~a, expected ~a"
                     (pipe-content-length rx-out) num-bytes-written 
(bytes-length bstr


(displayln "done")

 output:

avail: 16777216
space available? #t
pipe content length: 0
rx buffer overflow. pipe content length: 15, written 15, expected 509
done



That's ok ... my analysis was wrong anyhow.   It looks like the problem 
is that "flush" is not defined ... or simply doesn't work ... on 
Racket's pipes. *write-bytes-avail*  et al  depend on flush and so they 
won't work. *write-bytes*  works fine, but will block if you fill the pipe.


This looks like something for the development team.

George

--
You received this message because you are subscribed to the Google Groups "Racket 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/4836be0b-f785-9394-b18f-f5228970ba44%40comcast.net.

Re: [racket-users] Why would writing to a pipe fail?

[David Storrs]

Damn.  Sorry, I posted out of sync versions of code and output.  This is
correct:

(define bstr (make-shared-bytes 509 5))
(define rx-pipe-size 16777216)
(define-values (rx-in rx-out) (make-pipe rx-pipe-size))

(define (room-in-rx-pipe? bstr)
  (define avail (- rx-pipe-size (pipe-content-length rx-out)))
  (displayln (format "avail: ~a" avail))
  (<= (bytes-length bstr) avail))

(displayln (format "space available? ~a " (room-in-rx-pipe? bstr)))
(displayln (format "pipe content length: ~a" (pipe-content-length rx-out)))
(define num-bytes-written (write-bytes-avail* bstr rx-out))
(unless (eq? num-bytes-written (bytes-length bstr))
  (displayln (format "rx buffer overflow. pipe content length: ~a, written
~a, expected ~a"
 (pipe-content-length rx-out) num-bytes-written
(bytes-length bstr

(displayln "done")

 output:

avail: 16777216
space available? #t
pipe content length: 0
rx buffer overflow. pipe content length: 15, written 15, expected 509
done

On Wed, Jun 16, 2021 at 2:11 PM George Neuner  wrote:

>
> On 6/16/2021 1:19 PM, David Storrs wrote:
>
> I'm getting bytes off the wire and attempting to write them to a port.  I
> have a check in place to verify that the pipe has free space but when I
> attempt to reports that yes, there is space, and then it writes and fails
> regardless and I'm not sure why.  The following is a simplified version of
> the code but it produces the same results as the live version.
>
> I've dug through the docs on write-bytes-avail* and pipes and I'm
> scratching my head.  Any thoughts?
>
> --- code
> #lang racket
> (define bstr (make-shared-bytes 509 5))
> (define rx-pipe-size 16777216)
> (define-values (rx-in rx-out) (make-pipe rx-pipe-size))
>
> (define (room-in-rx-pipe? bstr)
>   (define avail (- rx-pipe-size (pipe-content-length rx-out)))
>   (displayln (format "space available: ~a" avail))
>   (<= (bytes-length bstr) avail))
>
> (displayln (format "space available? ~a " (room-in-rx-pipe? bstr)))
> (define num-bytes-written (write-bytes-avail* bstr rx-out))
> (unless (eq? num-bytes-written (bytes-length bstr))
>   (displayln (format "rx buffer overflow. num-bytes-written ~a, expected
> ~a"
>  num-bytes-written (bytes-length bstr
>
> (displayln "done")
>  /code
>
> The output is:
>
> --- output
> avail: 16777216
> space available? #t
> pipe content length: 0
> rx buffer overflow. pipe content length: 15, written 15, expected 509
> done
> --- /output
>
>
>
> I'm not sure what's going on either, but your output says the pipe
> contains 15 characters/bytes and THAT will cause *write-bytes-avail**  to
> fail ... it fails if there is existing buffered content.  I suspect that it
> will work if you substitute *write-bytes-avail*  (no *)  or have
> something read from the pipe before writing to it.
>
> I've never used Racket's pipes, but this looks very much like a connect
> handshake issue.
>
> George
>
> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-users/0619cc74-a15d-a0f2-38b7-09feeebf59c0%40comcast.net
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CAE8gKocz8BkvRc%2BeDhDt0WpOmqzkeEYdoQ9cfacGJhNkXkNugg%40mail.gmail.com.

Re: [racket-users] Why would writing to a pipe fail?

[George Neuner]

On 6/16/2021 1:19 PM, David Storrs wrote:
I'm getting bytes off the wire and attempting to write them to a 
port.  I have a check in place to verify that the pipe has free space 
but when I attempt to reports that yes, there is space, and then it 
writes and fails regardless and I'm not sure why.  The following is a 
simplified version of the code but it produces the same results as the 
live version.


I've dug through the docs on write-bytes-avail* and pipes and I'm 
scratching my head.  Any thoughts?


--- code
#lang racket
(define bstr (make-shared-bytes 509 5))
(define rx-pipe-size 16777216)
(define-values (rx-in rx-out) (make-pipe rx-pipe-size))

(define (room-in-rx-pipe? bstr)
  (define avail (- rx-pipe-size (pipe-content-length rx-out)))
  (displayln (format "space available: ~a" avail))
  (<= (bytes-length bstr) avail))

(displayln (format "space available? ~a " (room-in-rx-pipe? bstr)))
(define num-bytes-written (write-bytes-avail* bstr rx-out))
(unless (eq? num-bytes-written (bytes-length bstr))
  (displayln (format "rx buffer overflow. num-bytes-written ~a, 
expected ~a"

                     num-bytes-written (bytes-length bstr

(displayln "done")
 /code

The output is:

--- output
avail: 16777216
space available? #t
pipe content length: 0
rx buffer overflow. pipe content length: 15, written 15, expected 509
done
--- /output



I'm not sure what's going on either, but your output says the pipe 
contains 15 characters/bytes and THAT will cause *write-bytes-avail** to 
fail ... it fails if there is existing buffered content.  I suspect that 
it will work if you substitute *write-bytes-avail* (no *)  or have 
something read from the pipe before writing to it.


I've never used Racket's pipes, but this looks very much like a connect 
handshake issue.


George

--
You received this message because you are subscribed to the Google Groups "Racket 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/0619cc74-a15d-a0f2-38b7-09feeebf59c0%40comcast.net.