Hi Alistair,
I must take my part too: I suggested that we could use a pair of
getc/ungetc to know if we are atEnd(OfData), but this obviously works well
with AsyncFileIO only, else it blocks.
For files, this generally isn't a problem (but maybe for network mounted
partitions), the latency is bearable, but for sockets and pipes, that is
not the right thing.

2018-04-11 14:56 GMT+02:00 Alistair Grant <akgrant0...@gmail.com>:

> Hi Sven & Dennis,
>
> On 11 April 2018 at 12:04, Denis Kudriashov <dionisi...@gmail.com> wrote:
> > Hi Alistair.
> >
> > I don't think anybody is annoyed by you. You are doing really good job.
> And
> > nice thing that you are super patient to continue :)
> >
>
> On 11 April 2018 at 12:13, Sven Van Caekenberghe <s...@stfx.eu> wrote:
> >
> > Yes, Alistair, you are a top notch open source contributor !
> >
> > For me, this discussion is about the difference between looking from low
> level details/issues/changes up, vs, from a higher level down.
>
> Thanks for your kind words.
>
>
>
> > What I try to understand is why blocking atEnd is bad?
> > Here is code from VMMaker:
> >
> > [stdin atEnd] whileFalse:
> > [| nextChunk |
> > stdout nextPutAll: 'squeak> '; flush.
> > nextChunk := stdin nextChunkNoTag.
> > [nextChunk notEmpty and: [nextChunk first isSeparator]] whileTrue:
> > [nextChunk := nextChunk allButFirst].
> > Transcript cr; nextPutAll: nextChunk; cr; flush.
> > [stdout print: (Compiler evaluate: nextChunk); cr; flush]
> > on: Error
> > do: [:ex| self logError: ex description inContext: ex signalerContext to:
> > stderr]].
> > quitOnEof ifTrue:
> > [SourceFiles at: 2 put: nil.
> > Smalltalk snapshot: false andQuit: true]
> >
> >
> > I am not see why it breaks with blocking #atEnd. Can you explain?
>
>
> First consider the case where #atEnd doesn't block and we just want to
> evaluate 4+3:
>
> 1. #atEnd will return false
> 2. the loop will print the prompt
> 3. wait for input (stdin nextChunkNoTag)
> 4. print the result
> 5. goto 1.
>
> So the screen will look like:
>
> squeak> 4+3!
> 7
> squeak> [cursor here]
>
> Which is what we expect (prompt, input, result, prompt).
>
> If #atEnd is blocking the VM will hang at step 1 until the user enters
> something in the terminal.  In Ubuntu at least terminal input appears to
> be line buffered, so for the example above the terminal will look like:
>
> 4+3!
> squeak> 7
> [cursor here]
>
> We don't get the prompt when the program is started, the result is
> printed after the prompt, and then there's just a cursor sitting at the
> start of the next line.
>
> Obviously the program could be re-written to have the correct output
> with #atEnd blocking.  But I'm arguing that this program is
> representative of many others, and we don't want to break backward
> compatibility in this case.
>
> Cheers,
> Alistair
>
>
>
>
>
>
>
>
> > 2018-04-11 11:41 GMT+02:00 Alistair Grant <akgrant0...@gmail.com>:
> >>
> >> Hi Sven,
> >>
> >> Oh dear.  I feel as though I'm not getting my concerns across at all
> >> well, and I'm pushing hard enough that all I'm going to do is make
> >> people annoyed.  So let me try to restate the issue one last time
> >> before answering your questions directly.
> >>
> >> Pharo & Squeak have unwritten rules about stream usage that I suspect
> >> have just emerged over time without being designed.
> >>
> >> If you want to be able to iterate over any stream, and in particular
> >> stdin from a terminal (which, as far as I know, is the outlier that
> >> causes all the problems) you have to follow these rules:
> >>
> >> 1.  If the stream is character / byte oriented you have to check for
> >> EOF using "stream next == nil".  #atEnd can be used, but you'll still
> >> have to do the nil check.
> >>
> >> 2.  All other streams have to check for EOF (end of stream) using
> >> #atEnd.  "stream next == nil" can be used, but you'll still need to
> >> test #atEnd to determine whether nil is a value returned by the
> >> stream.
> >>
> >> If you write code that you want to be able to consume characters,
> >> bytes or any other object, you'll have to test both "stream next ==
> >> nil" and #atEnd.
> >>
> >> The rules are the result of the original blue book design being that
> >> #atEnd should be used, and then character input from a terminal being
> >> added later, but always returning an EOF character (nil) before #atEnd
> >> answers correctly.
> >>
> >> At the moment, ZnCharacterEncoder uses #atEnd on character / byte
> >> streams, so fails for stdin on a terminal.
> >>
> >> Back to your questions:
> >>
> >> On 11 April 2018 at 11:12, Sven Van Caekenberghe <s...@stfx.eu> wrote:
> >> >
> >> >
> >> >> On 11 Apr 2018, at 10:29, Alistair Grant <akgrant0...@gmail.com>
> wrote:
> >> >>
> >> >> Hi Denis,
> >> >>
> >> >> On 11 April 2018 at 10:02, Denis Kudriashov <dionisi...@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> 2018-04-11 8:32 GMT+02:00 Alistair Grant <akgrant0...@gmail.com>:
> >> >>>>
> >> >>>>>>> Where is it being said that #next and/or #atEnd should be
> blocking
> >> >>>>>>> or
> >> >>>>>>> non-blocking ?
> >> >>>>>>
> >> >>>>>> There is existing code that assumes that #atEnd is non-blocking
> and
> >> >>>>>> that #next is allowed block.  I believe that we should keep those
> >> >>>>>> conditions.
> >> >>>>>
> >> >>>>> I fail to see where that is written down, either way. Can you
> point
> >> >>>>> me
> >> >>>>> to comments stating that, I would really like to know ?
> >> >>>>
> >> >>>> I'm not aware of it being written down, just that ever existing
> >> >>>> implementation I'm aware of behaves this way.
> >> >>>>
> >> >>>> On the other hand, making #atEnd blocking breaks Eliot's REPL
> sample
> >> >>>> (in Squeak).
> >> >>>
> >> >>>
> >> >>> Could you write here this example, please?
> >> >>
> >> >> The code is loaded in squeak using:
> >> >>
> >> >>
> >> >> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/
> buildspurtrunkreaderimage.sh
> >> >>
> >> >> for 32 bit images.  It loads:
> >> >>
> >> >>
> >> >> https://github.com/OpenSmalltalk/opensmalltalk-
> vm/blob/Cog/image/LoadReader.st
> >> >>
> >> >> which loads package CogTools-Listener in
> >> >> http://source.squeak.org/VMMaker
> >> >>
> >> >> An image that automatically runs the code and nothing else is created
> >> >> in:
> >> >>
> >> >>
> >> >> https://github.com/OpenSmalltalk/opensmalltalk-
> vm/blob/Cog/image/StartReader.st
> >> >>
> >> >>
> >> >> If you want to run it interactively you can load CogTools-Listener
> and
> >> >> do something like:
> >> >>
> >> >> StdioListener new
> >> >>    quitOnEof: false;
> >> >>    run
> >> >
> >> > What does #quitOnEof: do ? Can the StdioListener code be
> browsed/viewed
> >> > online somewhere ?
> >>
> >> I just referenced this as an example of making #atEnd (really
> >> FilePlugin>>primitiveFileAtEnd) blocking causing problems.  I wasn't
> >> expecting people to go and look at the code or use it as a test.
> >>
> >> If you really want to look at it (from Pharo):
> >>
> >> 1. Add http://source.squeak.org/VMMaker as a repository.
> >> 2. Browse the CogTools-Listener package
> >>
> >>
> >> >> If you modify #atEnd to block it will result in the "squeak>" input
> >> >> prompt being printed in the terminal after the input has been
> entered.
> >> >
> >> > How does one modify #atEnd to block ? I suppose you are talking about
> >> > StdioStream>>#atEnd ?
> >>
> >> I meant the primitive, i.e. FilePlugin>>primitiveFileAtEnd /
> >> FilePluginPrims>>atEnd:.
> >>
> >>
> >> >  ^ self peek isNil
> >> >
> >> > ?
> >> >
> >> > PS: I liked your runnable example better, I will try it later on. Thx!
> >>
> >> Right.  My code is meant to be minimal and trigger the problem I'm
> >> actually focused on - that ZnCharacterEncoder doesn't work with stdin
> >> from a terminal.
> >>
> >> Sven has expressed a hesitation to change the internal operation of
> >> the Zinc streams from using #atEnd to "stream peek == nil" and this
> >> whole discussion is really about us trying to resolve our different
> >> perspective of the best path forward.  I respect Sven and his work so
> >> I'm trying to justify the change (but I'm not expressing it at all
> >> well, obviously).
> >>
> >> Cheers,
> >> Alistair
> >>
> >>
> >>
> >> >> The code can be loaded in to Pharo and basically works, but the
> output
> >> >> tends to be hidden behind the next input prompt because it uses #cr
> >> >> instead of #lf.  You can easily modify StdioListener>>initialize to
> >> >> set the line end convention in stdout.
> >> >>
> >> >> NOTE: It is not intended to be a release quality implementation of a
> >> >> evaluation loop.  The whole purpose as I understand it is for it to
> be
> >> >> as simple as possible to assist in tracking down issues using the VM
> >> >> simulator.  It runs minimal code to get to the point of waiting for
> >> >> user input and then allows an expression that causes problems to be
> >> >> entered and traced using the simulator.
> >> >>
> >> >> Cheers,
> >> >> Alistair
> >> >>
> >> >>
> >> >>
> >> >>>>>>> How is this related to how EOF is signalled ?
> >> >>>>>>
> >> >>>>>> Because, combined with terminal EOF not being known until the
> user
> >> >>>>>> explicitly flags it (with Ctrl-D) it means that #atEnd can't be
> >> >>>>>> used
> >> >>>>>> for iterating over input from stdin connected to a terminal.
> >> >>>>>
> >> >>>>> This seems to me like an exception that only holds for one
> >> >>>>> particular
> >> >>>>> stream in one particular scenario (interactive stdin). I might be
> >> >>>>> wrong.
> >> >>>>>
> >> >>>>>>> It seems to me that there are quite a few classes of streams
> that
> >> >>>>>>> are
> >> >>>>>>> 'special' in the sense that #next could be blocking and/or
> #atEnd
> >> >>>>>>> could be
> >> >>>>>>> unclear - socket/network streams, serial streams, maybe stdio
> >> >>>>>>> (interactive
> >> >>>>>>> or not). Without a message like #isDataAvailable you cannot
> handle
> >> >>>>>>> those
> >> >>>>>>> without blocking.
> >> >>>>>>
> >> >>>>>> Right.  I think this is a distraction (I was trying to explain
> some
> >> >>>>>> details, but it's causing more confusion instead of helping).
> >> >>>>>>
> >> >>>>>> The important point is that #atEnd doesn't work for iterating
> over
> >> >>>>>> streams with terminal input
> >> >>>>>
> >> >>>>> Maybe you should also point to the actual code that fails. I mean
> >> >>>>> you
> >> >>>>> showed a partial stack trace, but not how you got there,
> precisely.
> >> >>>>> How does
> >> >>>>> the application reading from an interactive stdin do to get into
> >> >>>>> trouble ?
> >> >>>>
> >> >>>> Included below.
> >> >>>>
> >> >>>>
> >> >>>>>>> Reading from stdin seems like a very rare case for a Smalltalk
> >> >>>>>>> system
> >> >>>>>>> (not that it should not be possible).
> >> >>>>>>
> >> >>>>>> There's been quite a bit of discussion and several projects
> >> >>>>>> recently
> >> >>>>>> related to using pharo for scripting, so it may become more
> common.
> >> >>>>>> E.g.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> https://www.quora.com/Can-Smalltalk-be-a-batch-file-
> scripting-language/answer/Philippe-Back-1?share=c19bfc95
> >> >>>>>> https://github.com/rajula96reddy/pharo-cli
> >> >>>>>
> >> >>>>> Still, it is not common at all.
> >> >>>>>
> >> >>>>>>> I have a feeling that too much functionality is being pushed
> into
> >> >>>>>>> too
> >> >>>>>>> small an API.
> >> >>>>>>
> >> >>>>>> This is just about how should Zinc streams be iterating over the
> >> >>>>>> underlying streams.  You didn't like checking the result of #next
> >> >>>>>> for
> >> >>>>>> nil since it isn't general, correctly pointing out that nil is a
> >> >>>>>> valid
> >> >>>>>> value for non-byte oriented streams.  But #atEnd doesn't work for
> >> >>>>>> stdin from a terminal.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> At this point I think there are three options:
> >> >>>>>>
> >> >>>>>> 1. Modify Zinc to check the return value of #next instead of
> using
> >> >>>>>> #atEnd.
> >> >>>>>>
> >> >>>>>> This is what all existing character / byte oriented streams in
> >> >>>>>> Squeak
> >> >>>>>> and Pharo do.  At that point the Zinc streams can be used on all
> >> >>>>>> file
> >> >>>>>> / stdio input and output.
> >> >>>>>
> >> >>>>> I agree that such code exists in many places, but there is lots of
> >> >>>>> stream reading that does not check for nils.
> >> >>>>
> >> >>>> Right.  Streams can be categorised in many ways, but for this
> >> >>>> discussion I think streams are broken in to two types:
> >> >>>>
> >> >>>> 1) Byte / Character oriented
> >> >>>> 2) All others
> >> >>>>
> >> >>>> For historical reasons, byte / character oriented streams need to
> >> >>>> check for EOF by using "stream next == nil" and all other streams
> >> >>>> should use #atEnd.
> >> >>>>
> >> >>>> This avoids the "nil being part of the domain" issue that was
> >> >>>> discussed earlier in the thread.
> >> >>>>
> >> >>>>
> >> >>>>>> 2. Modify all streams to signal EOF in some other way, i.e. a
> >> >>>>>> sentinel
> >> >>>>>> or notification / exception.
> >> >>>>>>
> >> >>>>>> This is what we were discussing below.  But it is a decent chunk
> of
> >> >>>>>> work with significant impact on the existing code base.
> >> >>>>>
> >> >>>>> Agreed. This would be a future extension.
> >> >>>>>
> >> >>>>>> 3. Require anyone who wants to read from stdin to code around
> >> >>>>>> Zinc's
> >> >>>>>> inability to handle terminal input.
> >> >>>>>>
> >> >>>>>> I'd prefer to avoid this option if possible.
> >> >>>>>
> >> >>>>> See higher for a more concrete usage example request.
> >> >>>>
> >> >>>>
> >> >>>> testAtEnd.st
> >> >>>> --
> >> >>>> | ch stream string stdin |
> >> >>>>
> >> >>>> 'stdio.cs' asFileReference fileIn.
> >> >>>> "stdin := FileStream stdin."
> >> >>>> stdin := ZnCharacterReadStream on:
> >> >>>>    (ZnBufferedReadStream on:
> >> >>>>        Stdio stdin).
> >> >>>> stream := (String new: 100) writeStream.
> >> >>>> ch := stdin next.
> >> >>>> [ ch == nil ] whileFalse: [
> >> >>>>    stream nextPut: ch.
> >> >>>>    ch := stdin next. ].
> >> >>>> string := stream contents.
> >> >>>> FileStream stdout
> >> >>>>    nextPutAll: string; lf;
> >> >>>>    nextPutAll: 'Characters read: ';
> >> >>>>    nextPutAll: string size asString;
> >> >>>>    lf.
> >> >>>> Smalltalk snapshot: false andQuit: true.
> >> >>>> --
> >> >>>>
> >> >>>> Execute with:
> >> >>>>
> >> >>>> ./pharo --headless Pharo7.0-64bit-e76f1a2.image testAtEnd.st
> >> >>>>
> >> >>>> and type Ctrl-D gives:
> >> >>>>
> >> >>>>
> >> >>>> 'Errors in script loaded from testAtEnd.st'
> >> >>>> MessageNotUnderstood: receiver of "<" is nil
> >> >>>> UndefinedObject(Object)>>doesNotUnderstand: #<
> >> >>>> ZnUTF8Encoder>>nextCodePointFromStream:
> >> >>>> ZnUTF8Encoder(ZnCharacterEncoder)>>nextFromStream:
> >> >>>> ZnCharacterReadStream>>nextElement
> >> >>>> ZnCharacterReadStream(ZnEncodedReadStream)>>next
> >> >>>> UndefinedObject>>DoIt
> >> >>>> OpalCompiler>>evaluate
> >> >>>>
> >> >>>>
> >> >>>> Using #atEnd to control the loop instead of "stdin next == nil"
> >> >>>> produces the same result.
> >> >>>>
> >> >>>> Replacing stdin with FileStream stdin makes the script work.
> >> >>>>
> >> >>>> stdio.cs fixes a bug in StdioStream which really isn't part of this
> >> >>>> discussion (PR to be submitted).
> >> >>>>
> >> >>>> Cheers,
> >> >>>> Alistair
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>>> Does that clarify the situation?
> >> >>>>>
> >> >>>>> Yes, it helps. Thanks. But questions remain.
> >> >>>>>
> >> >>>>>> Thanks,
> >> >>>>>> Alistair
>
>

Reply via email to