OS pipes are a similar case. On Pharo, you can run CommandShellTestCase to
provide some test coverage for this.

Dave

On Wed, Apr 11, 2018 at 03:13:35PM +0200, Denis Kudriashov wrote:
> Thanks for explanation.
> 
> I think it would be same scenario for socket stream where #atEnd is not
> blocking. So I agree that it is expected behaviour.
> 
> The example is general enough to expect it to be working for any given pair
> of in and out streams. So our streams should support this.
> 
> 
> 2018-04-11 14:56 GMT+02:00 Alistair Grant <akgrant0...@gmail.com>:
> 
> > Hi Sven & Dennis,
> >
> > On 11 April 2018 at 12:04, Denis Kudriashov <dionisi...@gmail.com> wrote:
> > > Hi Alistair.
> > >
> > > I don't think anybody is annoyed by you. You are doing really good job.
> > And
> > > nice thing that you are super patient to continue :)
> > >
> >
> > On 11 April 2018 at 12:13, Sven Van Caekenberghe <s...@stfx.eu> wrote:
> > >
> > > Yes, Alistair, you are a top notch open source contributor !
> > >
> > > For me, this discussion is about the difference between looking from low
> > level details/issues/changes up, vs, from a higher level down.
> >
> > Thanks for your kind words.
> >
> >
> >
> > > What I try to understand is why blocking atEnd is bad?
> > > Here is code from VMMaker:
> > >
> > > [stdin atEnd] whileFalse:
> > > [| nextChunk |
> > > stdout nextPutAll: 'squeak> '; flush.
> > > nextChunk := stdin nextChunkNoTag.
> > > [nextChunk notEmpty and: [nextChunk first isSeparator]] whileTrue:
> > > [nextChunk := nextChunk allButFirst].
> > > Transcript cr; nextPutAll: nextChunk; cr; flush.
> > > [stdout print: (Compiler evaluate: nextChunk); cr; flush]
> > > on: Error
> > > do: [:ex| self logError: ex description inContext: ex signalerContext to:
> > > stderr]].
> > > quitOnEof ifTrue:
> > > [SourceFiles at: 2 put: nil.
> > > Smalltalk snapshot: false andQuit: true]
> > >
> > >
> > > I am not see why it breaks with blocking #atEnd. Can you explain?
> >
> >
> > First consider the case where #atEnd doesn't block and we just want to
> > evaluate 4+3:
> >
> > 1. #atEnd will return false
> > 2. the loop will print the prompt
> > 3. wait for input (stdin nextChunkNoTag)
> > 4. print the result
> > 5. goto 1.
> >
> > So the screen will look like:
> >
> > squeak> 4+3!
> > 7
> > squeak> [cursor here]
> >
> > Which is what we expect (prompt, input, result, prompt).
> >
> > If #atEnd is blocking the VM will hang at step 1 until the user enters
> > something in the terminal.  In Ubuntu at least terminal input appears to
> > be line buffered, so for the example above the terminal will look like:
> >
> > 4+3!
> > squeak> 7
> > [cursor here]
> >
> > We don't get the prompt when the program is started, the result is
> > printed after the prompt, and then there's just a cursor sitting at the
> > start of the next line.
> >
> > Obviously the program could be re-written to have the correct output
> > with #atEnd blocking.  But I'm arguing that this program is
> > representative of many others, and we don't want to break backward
> > compatibility in this case.
> >
> > Cheers,
> > Alistair
> >
> >
> >
> >
> >
> >
> >
> >
> > > 2018-04-11 11:41 GMT+02:00 Alistair Grant <akgrant0...@gmail.com>:
> > >>
> > >> Hi Sven,
> > >>
> > >> Oh dear.  I feel as though I'm not getting my concerns across at all
> > >> well, and I'm pushing hard enough that all I'm going to do is make
> > >> people annoyed.  So let me try to restate the issue one last time
> > >> before answering your questions directly.
> > >>
> > >> Pharo & Squeak have unwritten rules about stream usage that I suspect
> > >> have just emerged over time without being designed.
> > >>
> > >> If you want to be able to iterate over any stream, and in particular
> > >> stdin from a terminal (which, as far as I know, is the outlier that
> > >> causes all the problems) you have to follow these rules:
> > >>
> > >> 1.  If the stream is character / byte oriented you have to check for
> > >> EOF using "stream next == nil".  #atEnd can be used, but you'll still
> > >> have to do the nil check.
> > >>
> > >> 2.  All other streams have to check for EOF (end of stream) using
> > >> #atEnd.  "stream next == nil" can be used, but you'll still need to
> > >> test #atEnd to determine whether nil is a value returned by the
> > >> stream.
> > >>
> > >> If you write code that you want to be able to consume characters,
> > >> bytes or any other object, you'll have to test both "stream next ==
> > >> nil" and #atEnd.
> > >>
> > >> The rules are the result of the original blue book design being that
> > >> #atEnd should be used, and then character input from a terminal being
> > >> added later, but always returning an EOF character (nil) before #atEnd
> > >> answers correctly.
> > >>
> > >> At the moment, ZnCharacterEncoder uses #atEnd on character / byte
> > >> streams, so fails for stdin on a terminal.
> > >>
> > >> Back to your questions:
> > >>
> > >> On 11 April 2018 at 11:12, Sven Van Caekenberghe <s...@stfx.eu> wrote:
> > >> >
> > >> >
> > >> >> On 11 Apr 2018, at 10:29, Alistair Grant <akgrant0...@gmail.com>
> > wrote:
> > >> >>
> > >> >> Hi Denis,
> > >> >>
> > >> >> On 11 April 2018 at 10:02, Denis Kudriashov <dionisi...@gmail.com>
> > >> >> wrote:
> > >> >>>
> > >> >>> 2018-04-11 8:32 GMT+02:00 Alistair Grant <akgrant0...@gmail.com>:
> > >> >>>>
> > >> >>>>>>> Where is it being said that #next and/or #atEnd should be
> > blocking
> > >> >>>>>>> or
> > >> >>>>>>> non-blocking ?
> > >> >>>>>>
> > >> >>>>>> There is existing code that assumes that #atEnd is non-blocking
> > and
> > >> >>>>>> that #next is allowed block.  I believe that we should keep those
> > >> >>>>>> conditions.
> > >> >>>>>
> > >> >>>>> I fail to see where that is written down, either way. Can you
> > point
> > >> >>>>> me
> > >> >>>>> to comments stating that, I would really like to know ?
> > >> >>>>
> > >> >>>> I'm not aware of it being written down, just that ever existing
> > >> >>>> implementation I'm aware of behaves this way.
> > >> >>>>
> > >> >>>> On the other hand, making #atEnd blocking breaks Eliot's REPL
> > sample
> > >> >>>> (in Squeak).
> > >> >>>
> > >> >>>
> > >> >>> Could you write here this example, please?
> > >> >>
> > >> >> The code is loaded in squeak using:
> > >> >>
> > >> >>
> > >> >> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/
> > buildspurtrunkreaderimage.sh
> > >> >>
> > >> >> for 32 bit images.  It loads:
> > >> >>
> > >> >>
> > >> >> https://github.com/OpenSmalltalk/opensmalltalk-
> > vm/blob/Cog/image/LoadReader.st
> > >> >>
> > >> >> which loads package CogTools-Listener in
> > >> >> http://source.squeak.org/VMMaker
> > >> >>
> > >> >> An image that automatically runs the code and nothing else is created
> > >> >> in:
> > >> >>
> > >> >>
> > >> >> https://github.com/OpenSmalltalk/opensmalltalk-
> > vm/blob/Cog/image/StartReader.st
> > >> >>
> > >> >>
> > >> >> If you want to run it interactively you can load CogTools-Listener
> > and
> > >> >> do something like:
> > >> >>
> > >> >> StdioListener new
> > >> >>    quitOnEof: false;
> > >> >>    run
> > >> >
> > >> > What does #quitOnEof: do ? Can the StdioListener code be
> > browsed/viewed
> > >> > online somewhere ?
> > >>
> > >> I just referenced this as an example of making #atEnd (really
> > >> FilePlugin>>primitiveFileAtEnd) blocking causing problems.  I wasn't
> > >> expecting people to go and look at the code or use it as a test.
> > >>
> > >> If you really want to look at it (from Pharo):
> > >>
> > >> 1. Add http://source.squeak.org/VMMaker as a repository.
> > >> 2. Browse the CogTools-Listener package
> > >>
> > >>
> > >> >> If you modify #atEnd to block it will result in the "squeak>" input
> > >> >> prompt being printed in the terminal after the input has been
> > entered.
> > >> >
> > >> > How does one modify #atEnd to block ? I suppose you are talking about
> > >> > StdioStream>>#atEnd ?
> > >>
> > >> I meant the primitive, i.e. FilePlugin>>primitiveFileAtEnd /
> > >> FilePluginPrims>>atEnd:.
> > >>
> > >>
> > >> >  ^ self peek isNil
> > >> >
> > >> > ?
> > >> >
> > >> > PS: I liked your runnable example better, I will try it later on. Thx!
> > >>
> > >> Right.  My code is meant to be minimal and trigger the problem I'm
> > >> actually focused on - that ZnCharacterEncoder doesn't work with stdin
> > >> from a terminal.
> > >>
> > >> Sven has expressed a hesitation to change the internal operation of
> > >> the Zinc streams from using #atEnd to "stream peek == nil" and this
> > >> whole discussion is really about us trying to resolve our different
> > >> perspective of the best path forward.  I respect Sven and his work so
> > >> I'm trying to justify the change (but I'm not expressing it at all
> > >> well, obviously).
> > >>
> > >> Cheers,
> > >> Alistair
> > >>
> > >>
> > >>
> > >> >> The code can be loaded in to Pharo and basically works, but the
> > output
> > >> >> tends to be hidden behind the next input prompt because it uses #cr
> > >> >> instead of #lf.  You can easily modify StdioListener>>initialize to
> > >> >> set the line end convention in stdout.
> > >> >>
> > >> >> NOTE: It is not intended to be a release quality implementation of a
> > >> >> evaluation loop.  The whole purpose as I understand it is for it to
> > be
> > >> >> as simple as possible to assist in tracking down issues using the VM
> > >> >> simulator.  It runs minimal code to get to the point of waiting for
> > >> >> user input and then allows an expression that causes problems to be
> > >> >> entered and traced using the simulator.
> > >> >>
> > >> >> Cheers,
> > >> >> Alistair
> > >> >>
> > >> >>
> > >> >>
> > >> >>>>>>> How is this related to how EOF is signalled ?
> > >> >>>>>>
> > >> >>>>>> Because, combined with terminal EOF not being known until the
> > user
> > >> >>>>>> explicitly flags it (with Ctrl-D) it means that #atEnd can't be
> > >> >>>>>> used
> > >> >>>>>> for iterating over input from stdin connected to a terminal.
> > >> >>>>>
> > >> >>>>> This seems to me like an exception that only holds for one
> > >> >>>>> particular
> > >> >>>>> stream in one particular scenario (interactive stdin). I might be
> > >> >>>>> wrong.
> > >> >>>>>
> > >> >>>>>>> It seems to me that there are quite a few classes of streams
> > that
> > >> >>>>>>> are
> > >> >>>>>>> 'special' in the sense that #next could be blocking and/or
> > #atEnd
> > >> >>>>>>> could be
> > >> >>>>>>> unclear - socket/network streams, serial streams, maybe stdio
> > >> >>>>>>> (interactive
> > >> >>>>>>> or not). Without a message like #isDataAvailable you cannot
> > handle
> > >> >>>>>>> those
> > >> >>>>>>> without blocking.
> > >> >>>>>>
> > >> >>>>>> Right.  I think this is a distraction (I was trying to explain
> > some
> > >> >>>>>> details, but it's causing more confusion instead of helping).
> > >> >>>>>>
> > >> >>>>>> The important point is that #atEnd doesn't work for iterating
> > over
> > >> >>>>>> streams with terminal input
> > >> >>>>>
> > >> >>>>> Maybe you should also point to the actual code that fails. I mean
> > >> >>>>> you
> > >> >>>>> showed a partial stack trace, but not how you got there,
> > precisely.
> > >> >>>>> How does
> > >> >>>>> the application reading from an interactive stdin do to get into
> > >> >>>>> trouble ?
> > >> >>>>
> > >> >>>> Included below.
> > >> >>>>
> > >> >>>>
> > >> >>>>>>> Reading from stdin seems like a very rare case for a Smalltalk
> > >> >>>>>>> system
> > >> >>>>>>> (not that it should not be possible).
> > >> >>>>>>
> > >> >>>>>> There's been quite a bit of discussion and several projects
> > >> >>>>>> recently
> > >> >>>>>> related to using pharo for scripting, so it may become more
> > common.
> > >> >>>>>> E.g.
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> https://www.quora.com/Can-Smalltalk-be-a-batch-file-
> > scripting-language/answer/Philippe-Back-1?share=c19bfc95
> > >> >>>>>> https://github.com/rajula96reddy/pharo-cli
> > >> >>>>>
> > >> >>>>> Still, it is not common at all.
> > >> >>>>>
> > >> >>>>>>> I have a feeling that too much functionality is being pushed
> > into
> > >> >>>>>>> too
> > >> >>>>>>> small an API.
> > >> >>>>>>
> > >> >>>>>> This is just about how should Zinc streams be iterating over the
> > >> >>>>>> underlying streams.  You didn't like checking the result of #next
> > >> >>>>>> for
> > >> >>>>>> nil since it isn't general, correctly pointing out that nil is a
> > >> >>>>>> valid
> > >> >>>>>> value for non-byte oriented streams.  But #atEnd doesn't work for
> > >> >>>>>> stdin from a terminal.
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> At this point I think there are three options:
> > >> >>>>>>
> > >> >>>>>> 1. Modify Zinc to check the return value of #next instead of
> > using
> > >> >>>>>> #atEnd.
> > >> >>>>>>
> > >> >>>>>> This is what all existing character / byte oriented streams in
> > >> >>>>>> Squeak
> > >> >>>>>> and Pharo do.  At that point the Zinc streams can be used on all
> > >> >>>>>> file
> > >> >>>>>> / stdio input and output.
> > >> >>>>>
> > >> >>>>> I agree that such code exists in many places, but there is lots of
> > >> >>>>> stream reading that does not check for nils.
> > >> >>>>
> > >> >>>> Right.  Streams can be categorised in many ways, but for this
> > >> >>>> discussion I think streams are broken in to two types:
> > >> >>>>
> > >> >>>> 1) Byte / Character oriented
> > >> >>>> 2) All others
> > >> >>>>
> > >> >>>> For historical reasons, byte / character oriented streams need to
> > >> >>>> check for EOF by using "stream next == nil" and all other streams
> > >> >>>> should use #atEnd.
> > >> >>>>
> > >> >>>> This avoids the "nil being part of the domain" issue that was
> > >> >>>> discussed earlier in the thread.
> > >> >>>>
> > >> >>>>
> > >> >>>>>> 2. Modify all streams to signal EOF in some other way, i.e. a
> > >> >>>>>> sentinel
> > >> >>>>>> or notification / exception.
> > >> >>>>>>
> > >> >>>>>> This is what we were discussing below.  But it is a decent chunk
> > of
> > >> >>>>>> work with significant impact on the existing code base.
> > >> >>>>>
> > >> >>>>> Agreed. This would be a future extension.
> > >> >>>>>
> > >> >>>>>> 3. Require anyone who wants to read from stdin to code around
> > >> >>>>>> Zinc's
> > >> >>>>>> inability to handle terminal input.
> > >> >>>>>>
> > >> >>>>>> I'd prefer to avoid this option if possible.
> > >> >>>>>
> > >> >>>>> See higher for a more concrete usage example request.
> > >> >>>>
> > >> >>>>
> > >> >>>> testAtEnd.st
> > >> >>>> --
> > >> >>>> | ch stream string stdin |
> > >> >>>>
> > >> >>>> 'stdio.cs' asFileReference fileIn.
> > >> >>>> "stdin := FileStream stdin."
> > >> >>>> stdin := ZnCharacterReadStream on:
> > >> >>>>    (ZnBufferedReadStream on:
> > >> >>>>        Stdio stdin).
> > >> >>>> stream := (String new: 100) writeStream.
> > >> >>>> ch := stdin next.
> > >> >>>> [ ch == nil ] whileFalse: [
> > >> >>>>    stream nextPut: ch.
> > >> >>>>    ch := stdin next. ].
> > >> >>>> string := stream contents.
> > >> >>>> FileStream stdout
> > >> >>>>    nextPutAll: string; lf;
> > >> >>>>    nextPutAll: 'Characters read: ';
> > >> >>>>    nextPutAll: string size asString;
> > >> >>>>    lf.
> > >> >>>> Smalltalk snapshot: false andQuit: true.
> > >> >>>> --
> > >> >>>>
> > >> >>>> Execute with:
> > >> >>>>
> > >> >>>> ./pharo --headless Pharo7.0-64bit-e76f1a2.image testAtEnd.st
> > >> >>>>
> > >> >>>> and type Ctrl-D gives:
> > >> >>>>
> > >> >>>>
> > >> >>>> 'Errors in script loaded from testAtEnd.st'
> > >> >>>> MessageNotUnderstood: receiver of "<" is nil
> > >> >>>> UndefinedObject(Object)>>doesNotUnderstand: #<
> > >> >>>> ZnUTF8Encoder>>nextCodePointFromStream:
> > >> >>>> ZnUTF8Encoder(ZnCharacterEncoder)>>nextFromStream:
> > >> >>>> ZnCharacterReadStream>>nextElement
> > >> >>>> ZnCharacterReadStream(ZnEncodedReadStream)>>next
> > >> >>>> UndefinedObject>>DoIt
> > >> >>>> OpalCompiler>>evaluate
> > >> >>>>
> > >> >>>>
> > >> >>>> Using #atEnd to control the loop instead of "stdin next == nil"
> > >> >>>> produces the same result.
> > >> >>>>
> > >> >>>> Replacing stdin with FileStream stdin makes the script work.
> > >> >>>>
> > >> >>>> stdio.cs fixes a bug in StdioStream which really isn't part of this
> > >> >>>> discussion (PR to be submitted).
> > >> >>>>
> > >> >>>> Cheers,
> > >> >>>> Alistair
> > >> >>>>
> > >> >>>>
> > >> >>>>
> > >> >>>>
> > >> >>>>>> Does that clarify the situation?
> > >> >>>>>
> > >> >>>>> Yes, it helps. Thanks. But questions remain.
> > >> >>>>>
> > >> >>>>>> Thanks,
> > >> >>>>>> Alistair
> >
> >

Reply via email to