Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Stephane Ducasse]

Thanks

On Thu, Apr 12, 2018 at 5:06 PM, Alistair Grant  wrote:
> On 11 April 2018 at 20:47, Sven Van Caekenberghe  wrote:
>>
>> But there is something wrong with what is returned by Stdio stdin
>>
>
> I've opened 
> https://pharo.fogbugz.com/f/cases/21692/StdioStream-incorrectly-delegates-atEnd-and-position-to-file
>
> PR in progress.
>
> Cheers,
> Alistair
>

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

On 11 April 2018 at 20:47, Sven Van Caekenberghe  wrote:
>
> But there is something wrong with what is returned by Stdio stdin
>

I've opened 
https://pharo.fogbugz.com/f/cases/21692/StdioStream-incorrectly-delegates-atEnd-and-position-to-file

PR in progress.

Cheers,
Alistair

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

On 12 April 2018 at 10:48, Alistair Grant  wrote:
>
> So I'm going to respectfully disagree and say that I think to raise a
> MNU instead of returning nil is wrong, and that changing the Zinc
> streams isn't such a big deal.

Sorry Sven, this sounds like I'm suggesting that your planning to
leave the MNU in there, which would be rather rude of me, but isn't
what I meant at all.

But I do think Zinc streams should handle #next returning nil before
#atEnd answers true.

Cheers,
Alistair

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

Hi Sven,

On Thu, Apr 12, 2018 at 09:23:21AM +0200, Sven Van Caekenberghe wrote:
>
>
> > On 12 Apr 2018, at 08:33, Alistair Grant  wrote:
> >
> > ...
> >
> > In your example you've carefully exited by some other means than
> > signalling end-of-file (Ctrl-D).
> >
> > I don't think we can reasonably say "if you write code with Zinc
> > streams, ensure that none of your users ever press Ctrl-D, they must
> > always type 'quit' (or whatever)".
>
> I left that open as an exercise for the reader - I knew you would try that ;-)

Glad that I didn't disappoint. :-)


> The code was originally written for network streams and there it
> catches ConnectionClosed which is the network stream equivalent of EOF
> (more or less).
>
> I think we need something like an EndOfStream exception which should
> be thrown in that case, instead of returning nil.
>
> Returning nil from ZnCharacterReadStream or ZnCharacterEncoder when
> the stream is EOF is wrong IMHO. The stream being EOF is an
> exceptional situation (even though it is relatively common) and there
> is always #atEnd that works well in 99% of the cases.
>
> AFAIK that is also the design decision taking by Xtreams (where I
> believe the Incomplete exception replaces #atEnd).

OK, this brings us full-circle.

Earlier in the thread we wrote:


On Tue, Apr 10, 2018 at 11:54:39PM +0200, Sven Van Caekenberghe wrote:
> Alistair,
>
> I am replying in-between, please don't take my remarks as being
> negative, I just like to straighten things out as clear as possible.
>
> > On 10 Apr 2018, at 22:14, Alistair Grant  wrote:
> >
> > At this point I think there are three options:
> >
> > 1. Modify Zinc to check the return value of #next instead of using #atEnd.
> >
> > This is what all existing character / byte oriented streams in Squeak
> > and Pharo do.  At that point the Zinc streams can be used on all file
> > / stdio input and output.
>
> I agree that such code exists in many places, but there is lots of
> stream reading that does not check for nils.
>
> > 2. Modify all streams to signal EOF in some other way, i.e. a sentinel
> > or notification / exception.
> >
> > This is what we were discussing below.  But it is a decent chunk of
> > work with significant impact on the existing code base.
>
> Agreed. This would be a future extension.


I agree that raising an exception is a nicer solution (although for
anyone who disagrees I'll also admit that there is a subjective element to
that assessment).

But the existing behaviour of returning nil on EOF has been around for a
long time and works reasonably well, and as we agreed here, changing that
behaviour will have a significant impact on the existing code base.

None of this excludes Zinc streams adopting exceptions at any time in
the future.

So I'm going to respectfully disagree and say that I think to raise a
MNU instead of returning nil is wrong, and that changing the Zinc
streams isn't such a big deal.

Cheers,
Alistair

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

> On 12 Apr 2018, at 03:15, Eliot Miranda  wrote:
> 
> Hi Sven,
> 
> On Wed, Apr 11, 2018 at 1:25 PM, Sven Van Caekenberghe  wrote:
> 
> 
> > On 11 Apr 2018, at 21:44, Stephane Ducasse  wrote:
> >
> > I did not know about the NeoConsole. Nice because I wanted to build a
> > little REPL for my minilanguage implementation.
> 
> You are of course welcome to look at it.
> But it is Pharo specific.
> I use it to be able to hook/look into running headless server images.
> For this it is super handy.
> 
> Cool usage!  Could you tell me whether you type Smalltalk expressions into 
> this to examine your running server?  The answer will be used in a related 
> discussion on a mailing list not too distant from this one ;-)

Yes of course, that it the whole purpose, to type in expressions and to 
manipulate objects in a running image. I recently added options to look at code 
and change/add methods. All very primitive, but when in trouble, it works well.

> > Stef
> >
> > On Wed, Apr 11, 2018 at 8:47 PM, Sven Van Caekenberghe  wrote:
> >> Alistair,
> >>
> >>> On 11 Apr 2018, at 19:42, Sven Van Caekenberghe  wrote:
> >>>
> >>> I will send you some code later on.
> >>
> >> Today I arranged for my NeoConsole code (that normally works over a 
> >> network connection) to work over stdio. Although I am not yet happy with 
> >> every aspect of the implementation, it does work (using unaltered Zn 
> >> streams and code). The foll
> >>
> >> $ cat /etc/issue
> >> Ubuntu 16.04.4 LTS
> >> $ mkdir pharo7
> >> $ cd pharo7/
> >> $ curl get.pharo.org/70+vm | bash
> >>  % Total% Received % Xferd  Average Speed   TimeTime Time  
> >> Current
> >> Dload  Upload   Total   SpentLeft  
> >> Speed
> >> 100  3036  100  30360 0  36799  0 --:--:-- --:--:-- --:--:-- 
> >> 37024
> >> Downloading the latest 70 Image:
> >>http://files.pharo.org/get-files/70/pharo.zip
> >> Pharo.image
> >> Downloading the latest pharoVM:
> >>http://files.pharo.org/get-files/70/pharo-linux-stable.zip
> >> pharo-vm/pharo
> >> Creating starter scripts pharo and pharo-ui
> >> On a 64-bit system? You must enable and install the 32-bit libraries
> >>   Please see http://pharo.org/gnu-linux-installation for detailed 
> >> instructions
> >> $ ./pharo Pharo.image config http://mc.stfx.eu/Neo 
> >> ConfigurationOfNeoConsole --install=bleedingEdge
> >> 'Installing ConfigurationOfNeoConsole bleedingEdge'
> >>
> >> Loading 1-baseline of ConfigurationOfNeoConsole...
> >> Fetched -> Neo-Console-Core-SvenVanCaekenberghe.24 --- 
> >> http://mc.stfx.eu/Neo --- http://mc.stfx.eu/Neo
> >> Loaded -> Neo-Console-Core-SvenVanCaekenberghe.24 --- 
> >> http://mc.stfx.eu/Neo --- cache
> >> ...finished 1-baseline
> >> $ ./pharo Pharo.image eval NeoConsoleStdio run
> >> Neo Console 
> >> Pharo-7.0+alpha.build.760.sha.d2734dcabda799803c307365bcd120f92211d34a (32 
> >> Bit)
> >> pharo> 1+2
> >>
> >> 3
> >> pharo> 42 factorial
> >>
> >> 14050061177528798985431426062445115699363840
> >> pharo> Stdio stdin
> >>
> >> StdioStream: #stdin
> >> pharo> ==
> >> self: StdioStream: #stdin
> >> class: StdioStream
> >> file: a File
> >> handle: #[148 213 25 107 160 197 105 247 0 0 0 0 0 0 0 0 0 1 255 1]
> >> forWrite: false
> >> peekBuffer: nil
> >> pharo> show StdioStream>>#atEnd
> >> StdioStream>>#atEnd
> >> atEnd
> >>
> >>^ file atEnd
> >> pharo> get process.list
> >> Morphic UI Process
> >> Delay Scheduling Process
> >> Low Space Watcher
> >> Input Event Fetcher Process
> >> Idle Process
> >> WeakArray Finalization Process
> >> CommandLine handler process
> >> pharo> quit
> >> Bye!
> >> a NeoConsoleStdio
> >>
> >> I know there are many approaches to a REPL, I don't claim mine is best, it 
> >> is just the one that I have been using for years.
> >>
> >> In the above, I do not depend on EOF - just to be clear. The point being 
> >> that there is no immediate fundamental problem.
> >>
> >> But there is something wrong with what is returned by Stdio stdin
> >>
> >> Sven
> >>
> >>
> >>
> >
> 
> 
> 
> 
> 
> -- 
> _,,,^..^,,,_
> best, Eliot

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

> On 12 Apr 2018, at 04:07, Ben Coman  wrote:
> 
> 
> 
> On 12 April 2018 at 04:25, Sven Van Caekenberghe  wrote:
> 
> 
> > On 11 Apr 2018, at 21:44, Stephane Ducasse  wrote:
> >
> > I did not know about the NeoConsole. Nice because I wanted to build a
> > little REPL for my minilanguage implementation.
> 
> You are of course welcome to look at it.
> But it is Pharo specific.
> I use it to be able to hook/look into running headless server images.
> For this it is super handy.
> 
> A description of how to set that up would make a nice blog post.

It is part of, and partially documented in 
https://github.com/svenvc/pharo-server-tools

But maybe I'll write something about it, yes.

> cheers -ben 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

> On 12 Apr 2018, at 08:33, Alistair Grant  wrote:
> 
> Hi Sven,
> 
> On 11 April 2018 at 20:47, Sven Van Caekenberghe  wrote:
>> Alistair,
>> 
>>> On 11 Apr 2018, at 19:42, Sven Van Caekenberghe  wrote:
>>> 
>>> I will send you some code later on.
>> 
>> Today I arranged for my NeoConsole code (that normally works over a network 
>> connection) to work over stdio. Although I am not yet happy with every 
>> aspect of the implementation, it does work (using unaltered Zn streams and 
>> code). The foll
>> 
>> $ cat /etc/issue
>> Ubuntu 16.04.4 LTS
>> $ mkdir pharo7
>> $ cd pharo7/
>> $ curl get.pharo.org/70+vm | bash
>>  % Total% Received % Xferd  Average Speed   TimeTime Time  
>> Current
>> Dload  Upload   Total   SpentLeft  Speed
>> 100  3036  100  30360 0  36799  0 --:--:-- --:--:-- --:--:-- 
>> 37024
>> Downloading the latest 70 Image:
>>http://files.pharo.org/get-files/70/pharo.zip
>> Pharo.image
>> Downloading the latest pharoVM:
>>http://files.pharo.org/get-files/70/pharo-linux-stable.zip
>> pharo-vm/pharo
>> Creating starter scripts pharo and pharo-ui
>> On a 64-bit system? You must enable and install the 32-bit libraries
>>   Please see http://pharo.org/gnu-linux-installation for detailed 
>> instructions
>> $ ./pharo Pharo.image config http://mc.stfx.eu/Neo ConfigurationOfNeoConsole 
>> --install=bleedingEdge
>> 'Installing ConfigurationOfNeoConsole bleedingEdge'
>> 
>> Loading 1-baseline of ConfigurationOfNeoConsole...
>> Fetched -> Neo-Console-Core-SvenVanCaekenberghe.24 --- http://mc.stfx.eu/Neo 
>> --- http://mc.stfx.eu/Neo
>> Loaded -> Neo-Console-Core-SvenVanCaekenberghe.24 --- http://mc.stfx.eu/Neo 
>> --- cache
>> ...finished 1-baseline
>> $ ./pharo Pharo.image eval NeoConsoleStdio run
>> Neo Console 
>> Pharo-7.0+alpha.build.760.sha.d2734dcabda799803c307365bcd120f92211d34a (32 
>> Bit)
>> pharo> 1+2
>> 
>> 3
>> pharo> 42 factorial
>> 
>> 14050061177528798985431426062445115699363840
>> pharo> Stdio stdin
>> 
>> StdioStream: #stdin
>> pharo> ==
>> self: StdioStream: #stdin
>> class: StdioStream
>> file: a File
>> handle: #[148 213 25 107 160 197 105 247 0 0 0 0 0 0 0 0 0 1 255 1]
>> forWrite: false
>> peekBuffer: nil
>> pharo> show StdioStream>>#atEnd
>> StdioStream>>#atEnd
>> atEnd
>> 
>>^ file atEnd
>> pharo> get process.list
>> Morphic UI Process
>> Delay Scheduling Process
>> Low Space Watcher
>> Input Event Fetcher Process
>> Idle Process
>> WeakArray Finalization Process
>> CommandLine handler process
>> pharo> quit
>> Bye!
>> a NeoConsoleStdio
>> 
>> I know there are many approaches to a REPL, I don't claim mine is best, it 
>> is just the one that I have been using for years.
>> 
>> In the above, I do not depend on EOF - just to be clear. The point being 
>> that there is no immediate fundamental problem.
> 
> 
> Cool.  I've wanted to try NeoConsole for a while (I've got a Raspberry
> Pi monitoring temperatures and wanted to use it to work on the Pi) but
> hadn't made the time.
> 
> But... :-)
> 
> 
> $ vm/pharo Pharo7.0-64bit-d2734dc.image eval NeoConsoleStdio run
> Neo Console 
> Pharo-7.0+alpha.build.760.sha.d2734dcabda799803c307365bcd120f92211d34a
> (64 Bit)
> pharo> 4+3
> 
> 7
> pharo> [Ctrl-D]
> MessageNotUnderstood: receiver of "<" is nil
> UndefinedObject(Object)>>doesNotUnderstand: #<
> ZnUTF8Encoder>>nextCodePointFromStream:
> ZnUTF8Encoder(ZnCharacterEncoder)>>nextFromStream:
> ZnCharacterReadStream>>nextElement
> ZnCharacterReadStream(ZnEncodedReadStream)>>next
> [ :out |
> | eol char |
> eol := false.
> [ eol ]
> whileFalse: [ char := readStream next.
> (char isNil or: [ char == lf ])
> ifTrue: [ eol := true ]
> ifFalse: [ char == cr
> ifTrue: [ eol := true.
> readStream peekFor: lf ]
> ifFalse: [ out nextPut: char ] ] ] ] in ZnFastLineReader>>nextLine in
> Block: [ :out | ...
> ZnFastLineReader>>streamContents:
> ZnFastLineReader>>nextLine
> [ lineReader nextLine trimBoth ] in NeoConsoleStdio>>readInputFrom: in
> Block: [ lineReader nextLine trimBoth ]
> BlockClosure>>on:do:
> NeoConsoleStdio>>readInputFrom:
> NeoConsoleStdio>>executeRequestResponseLoop
> NeoConsoleStdio>>run
> NeoConsoleStdio class>>run
> UndefinedObject>>DoIt
> 
> 
> In your example you've carefully exited by some other means than
> signalling end-of-file (Ctrl-D).
> 
> I don't think we can reasonably say "if you write code with Zinc
> streams, ensure that none of your users ever press Ctrl-D, they must
> always type 'quit' (or whatever)".

I left that open as an exercise for the reader - I knew you would try that ;-)

The code was originally written for network streams and there it catches 
ConnectionClosed which is the network stream equivalent of EOF (more or less).

I think we need something like an EndOfStream exception which should be thrown 
in that case, instead of returning nil.

Returning nil from ZnCharacterReadStream or ZnCharacterEncoder when the 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

Hi Sven,

On 11 April 2018 at 20:47, Sven Van Caekenberghe  wrote:
> Alistair,
>
>> On 11 Apr 2018, at 19:42, Sven Van Caekenberghe  wrote:
>>
>> I will send you some code later on.
>
> Today I arranged for my NeoConsole code (that normally works over a network 
> connection) to work over stdio. Although I am not yet happy with every aspect 
> of the implementation, it does work (using unaltered Zn streams and code). 
> The foll
>
> $ cat /etc/issue
> Ubuntu 16.04.4 LTS
> $ mkdir pharo7
> $ cd pharo7/
> $ curl get.pharo.org/70+vm | bash
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
> 100  3036  100  30360 0  36799  0 --:--:-- --:--:-- --:--:-- 37024
> Downloading the latest 70 Image:
> http://files.pharo.org/get-files/70/pharo.zip
> Pharo.image
> Downloading the latest pharoVM:
> http://files.pharo.org/get-files/70/pharo-linux-stable.zip
> pharo-vm/pharo
> Creating starter scripts pharo and pharo-ui
> On a 64-bit system? You must enable and install the 32-bit libraries
>Please see http://pharo.org/gnu-linux-installation for detailed 
> instructions
> $ ./pharo Pharo.image config http://mc.stfx.eu/Neo ConfigurationOfNeoConsole 
> --install=bleedingEdge
> 'Installing ConfigurationOfNeoConsole bleedingEdge'
>
> Loading 1-baseline of ConfigurationOfNeoConsole...
> Fetched -> Neo-Console-Core-SvenVanCaekenberghe.24 --- http://mc.stfx.eu/Neo 
> --- http://mc.stfx.eu/Neo
> Loaded -> Neo-Console-Core-SvenVanCaekenberghe.24 --- http://mc.stfx.eu/Neo 
> --- cache
> ...finished 1-baseline
> $ ./pharo Pharo.image eval NeoConsoleStdio run
> Neo Console 
> Pharo-7.0+alpha.build.760.sha.d2734dcabda799803c307365bcd120f92211d34a (32 
> Bit)
> pharo> 1+2
>
> 3
> pharo> 42 factorial
>
> 14050061177528798985431426062445115699363840
> pharo> Stdio stdin
>
> StdioStream: #stdin
> pharo> ==
> self: StdioStream: #stdin
> class: StdioStream
> file: a File
> handle: #[148 213 25 107 160 197 105 247 0 0 0 0 0 0 0 0 0 1 255 1]
> forWrite: false
> peekBuffer: nil
> pharo> show StdioStream>>#atEnd
> StdioStream>>#atEnd
> atEnd
>
> ^ file atEnd
> pharo> get process.list
> Morphic UI Process
> Delay Scheduling Process
> Low Space Watcher
> Input Event Fetcher Process
> Idle Process
> WeakArray Finalization Process
> CommandLine handler process
> pharo> quit
> Bye!
> a NeoConsoleStdio
>
> I know there are many approaches to a REPL, I don't claim mine is best, it is 
> just the one that I have been using for years.
>
> In the above, I do not depend on EOF - just to be clear. The point being that 
> there is no immediate fundamental problem.


Cool.  I've wanted to try NeoConsole for a while (I've got a Raspberry
Pi monitoring temperatures and wanted to use it to work on the Pi) but
hadn't made the time.

But... :-)


$ vm/pharo Pharo7.0-64bit-d2734dc.image eval NeoConsoleStdio run
Neo Console 
Pharo-7.0+alpha.build.760.sha.d2734dcabda799803c307365bcd120f92211d34a
(64 Bit)
pharo> 4+3

7
pharo> [Ctrl-D]
MessageNotUnderstood: receiver of "<" is nil
UndefinedObject(Object)>>doesNotUnderstand: #<
ZnUTF8Encoder>>nextCodePointFromStream:
ZnUTF8Encoder(ZnCharacterEncoder)>>nextFromStream:
ZnCharacterReadStream>>nextElement
ZnCharacterReadStream(ZnEncodedReadStream)>>next
[ :out |
| eol char |
eol := false.
[ eol ]
whileFalse: [ char := readStream next.
(char isNil or: [ char == lf ])
ifTrue: [ eol := true ]
ifFalse: [ char == cr
ifTrue: [ eol := true.
readStream peekFor: lf ]
ifFalse: [ out nextPut: char ] ] ] ] in ZnFastLineReader>>nextLine in
Block: [ :out | ...
ZnFastLineReader>>streamContents:
ZnFastLineReader>>nextLine
[ lineReader nextLine trimBoth ] in NeoConsoleStdio>>readInputFrom: in
Block: [ lineReader nextLine trimBoth ]
BlockClosure>>on:do:
NeoConsoleStdio>>readInputFrom:
NeoConsoleStdio>>executeRequestResponseLoop
NeoConsoleStdio>>run
NeoConsoleStdio class>>run
UndefinedObject>>DoIt


In your example you've carefully exited by some other means than
signalling end-of-file (Ctrl-D).

I don't think we can reasonably say "if you write code with Zinc
streams, ensure that none of your users ever press Ctrl-D, they must
always type 'quit' (or whatever)".



>
> But there is something wrong with what is returned by Stdio stdin

Right, that's the bug I was addressing stdio.cs.  I've got a note to
open an issue and submit a PR.

Cheers,
Alistair

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[K K Subbu]

On Wednesday 11 April 2018 10:38 PM, Alistair Grant wrote:

StandardFileStream>>readInto:startingAt:count: assumes that
primitiveFileRead will always attempt to read count bytes, but it
actually only attempts to read 1.


StandardFileStream>>#basicNext uses

  position < readLimit ifFalse:

and does not check if the primRead returns < 0. Instead it should call 
whileFalse: and check if primRead returns <= 0 to check for EOF or error.


#primRead:... returns an error if the primitive fails and the file is 
closed. It should instead check for EOF or error and then return -1, 
otherwise it should return 0 (Data not ready?). Raise an error only if 
the underlying primitive fails for non-EOF cases.


Regards .. Subbu

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

On 11 April 2018 at 19:42, Nicolas Cellier
 wrote:
>
>
> 2018-04-11 19:08 GMT+02:00 Alistair Grant :
>>
>> Hi Sven,
>>
>> On 11 April 2018 at 18:53, Sven Van Caekenberghe  wrote:
>> > Something is off (and/or I am getting crazy, probably both).
>> >
>> > $ ./pharo --headless Pharo.image eval '(FileStream stdin binary; next:
>> > 3)'
>> > 123
>> > #[49]
>> >
>> > ??
>> >
>> > This should return #[49 50 51] AFAIK.
>>
>> I haven't checked properly, but I can take a guess at this...
>>
>> The file read primitive in FilePlugin tries to read from stdin 1
>> character at a time (presumably to get around the line buffering that
>> is done, but I haven't gone through the history.  At one time I think
>> it modified the terminal settings, but that code is #if'd out).  For
>> all other streams it attempts to read the requested number of
>> characters.
>>
>> StandardFileStream>>readInto:startingAt:count: assumes that
>> primitiveFileRead will always attempt to read count bytes, but it
>> actually only attempts to read 1.
>>
>> Cheers,
>> Alistair
>>
>
> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/150

Thanks!  I'd already looked at this routine and found it strangely
constructed.  Having some background discussion will help a lot.

Cheers,
Alistair

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Ben Coman]

On 12 April 2018 at 04:25, Sven Van Caekenberghe  wrote:

>
>
> > On 11 Apr 2018, at 21:44, Stephane Ducasse 
> wrote:
> >
> > I did not know about the NeoConsole. Nice because I wanted to build a
> > little REPL for my minilanguage implementation.
>
> You are of course welcome to look at it.
> But it is Pharo specific.
> I use it to be able to hook/look into running headless server images.
> For this it is super handy.


A description of how to set that up would make a nice blog post.
cheers -ben

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Eliot Miranda]

Hi Sven,

On Wed, Apr 11, 2018 at 1:25 PM, Sven Van Caekenberghe  wrote:

>
>
> > On 11 Apr 2018, at 21:44, Stephane Ducasse 
> wrote:
> >
> > I did not know about the NeoConsole. Nice because I wanted to build a
> > little REPL for my minilanguage implementation.
>
> You are of course welcome to look at it.
> But it is Pharo specific.
> I use it to be able to hook/look into running headless server images.
> For this it is super handy.
>

Cool usage!  Could you tell me whether you type Smalltalk expressions into
this to examine your running server?  The answer will be used in a related
discussion on a mailing list not too distant from this one ;-)


>
> > Stef
> >
> > On Wed, Apr 11, 2018 at 8:47 PM, Sven Van Caekenberghe 
> wrote:
> >> Alistair,
> >>
> >>> On 11 Apr 2018, at 19:42, Sven Van Caekenberghe  wrote:
> >>>
> >>> I will send you some code later on.
> >>
> >> Today I arranged for my NeoConsole code (that normally works over a
> network connection) to work over stdio. Although I am not yet happy with
> every aspect of the implementation, it does work (using unaltered Zn
> streams and code). The foll
> >>
> >> $ cat /etc/issue
> >> Ubuntu 16.04.4 LTS
> >> $ mkdir pharo7
> >> $ cd pharo7/
> >> $ curl get.pharo.org/70+vm | bash
> >>  % Total% Received % Xferd  Average Speed   TimeTime Time
> Current
> >> Dload  Upload   Total   SpentLeft
> Speed
> >> 100  3036  100  30360 0  36799  0 --:--:-- --:--:--
> --:--:-- 37024
> >> Downloading the latest 70 Image:
> >>http://files.pharo.org/get-files/70/pharo.zip
> >> Pharo.image
> >> Downloading the latest pharoVM:
> >>http://files.pharo.org/get-files/70/pharo-linux-stable.zip
> >> pharo-vm/pharo
> >> Creating starter scripts pharo and pharo-ui
> >> On a 64-bit system? You must enable and install the 32-bit libraries
> >>   Please see http://pharo.org/gnu-linux-installation for detailed
> instructions
> >> $ ./pharo Pharo.image config http://mc.stfx.eu/Neo
> ConfigurationOfNeoConsole --install=bleedingEdge
> >> 'Installing ConfigurationOfNeoConsole bleedingEdge'
> >>
> >> Loading 1-baseline of ConfigurationOfNeoConsole...
> >> Fetched -> Neo-Console-Core-SvenVanCaekenberghe.24 ---
> http://mc.stfx.eu/Neo --- http://mc.stfx.eu/Neo
> >> Loaded -> Neo-Console-Core-SvenVanCaekenberghe.24 ---
> http://mc.stfx.eu/Neo --- cache
> >> ...finished 1-baseline
> >> $ ./pharo Pharo.image eval NeoConsoleStdio run
> >> Neo Console Pharo-7.0+alpha.build.760.sha.
> d2734dcabda799803c307365bcd120f92211d34a (32 Bit)
> >> pharo> 1+2
> >>
> >> 3
> >> pharo> 42 factorial
> >>
> >> 14050061177528798985431426062445115699363840
> >> pharo> Stdio stdin
> >>
> >> StdioStream: #stdin
> >> pharo> ==
> >> self: StdioStream: #stdin
> >> class: StdioStream
> >> file: a File
> >> handle: #[148 213 25 107 160 197 105 247 0 0 0 0 0 0 0 0 0 1 255 1]
> >> forWrite: false
> >> peekBuffer: nil
> >> pharo> show StdioStream>>#atEnd
> >> StdioStream>>#atEnd
> >> atEnd
> >>
> >>^ file atEnd
> >> pharo> get process.list
> >> Morphic UI Process
> >> Delay Scheduling Process
> >> Low Space Watcher
> >> Input Event Fetcher Process
> >> Idle Process
> >> WeakArray Finalization Process
> >> CommandLine handler process
> >> pharo> quit
> >> Bye!
> >> a NeoConsoleStdio
> >>
> >> I know there are many approaches to a REPL, I don't claim mine is best,
> it is just the one that I have been using for years.
> >>
> >> In the above, I do not depend on EOF - just to be clear. The point
> being that there is no immediate fundamental problem.
> >>
> >> But there is something wrong with what is returned by Stdio stdin
> >>
> >> Sven
> >>
> >>
> >>
> >
>
>
>


-- 
_,,,^..^,,,_
best, Eliot

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

> On 11 Apr 2018, at 21:44, Stephane Ducasse  wrote:
> 
> I did not know about the NeoConsole. Nice because I wanted to build a
> little REPL for my minilanguage implementation.

You are of course welcome to look at it.
But it is Pharo specific.
I use it to be able to hook/look into running headless server images.
For this it is super handy.

> Stef
> 
> On Wed, Apr 11, 2018 at 8:47 PM, Sven Van Caekenberghe  wrote:
>> Alistair,
>> 
>>> On 11 Apr 2018, at 19:42, Sven Van Caekenberghe  wrote:
>>> 
>>> I will send you some code later on.
>> 
>> Today I arranged for my NeoConsole code (that normally works over a network 
>> connection) to work over stdio. Although I am not yet happy with every 
>> aspect of the implementation, it does work (using unaltered Zn streams and 
>> code). The foll
>> 
>> $ cat /etc/issue
>> Ubuntu 16.04.4 LTS
>> $ mkdir pharo7
>> $ cd pharo7/
>> $ curl get.pharo.org/70+vm | bash
>>  % Total% Received % Xferd  Average Speed   TimeTime Time  
>> Current
>> Dload  Upload   Total   SpentLeft  Speed
>> 100  3036  100  30360 0  36799  0 --:--:-- --:--:-- --:--:-- 
>> 37024
>> Downloading the latest 70 Image:
>>http://files.pharo.org/get-files/70/pharo.zip
>> Pharo.image
>> Downloading the latest pharoVM:
>>http://files.pharo.org/get-files/70/pharo-linux-stable.zip
>> pharo-vm/pharo
>> Creating starter scripts pharo and pharo-ui
>> On a 64-bit system? You must enable and install the 32-bit libraries
>>   Please see http://pharo.org/gnu-linux-installation for detailed 
>> instructions
>> $ ./pharo Pharo.image config http://mc.stfx.eu/Neo ConfigurationOfNeoConsole 
>> --install=bleedingEdge
>> 'Installing ConfigurationOfNeoConsole bleedingEdge'
>> 
>> Loading 1-baseline of ConfigurationOfNeoConsole...
>> Fetched -> Neo-Console-Core-SvenVanCaekenberghe.24 --- http://mc.stfx.eu/Neo 
>> --- http://mc.stfx.eu/Neo
>> Loaded -> Neo-Console-Core-SvenVanCaekenberghe.24 --- http://mc.stfx.eu/Neo 
>> --- cache
>> ...finished 1-baseline
>> $ ./pharo Pharo.image eval NeoConsoleStdio run
>> Neo Console 
>> Pharo-7.0+alpha.build.760.sha.d2734dcabda799803c307365bcd120f92211d34a (32 
>> Bit)
>> pharo> 1+2
>> 
>> 3
>> pharo> 42 factorial
>> 
>> 14050061177528798985431426062445115699363840
>> pharo> Stdio stdin
>> 
>> StdioStream: #stdin
>> pharo> ==
>> self: StdioStream: #stdin
>> class: StdioStream
>> file: a File
>> handle: #[148 213 25 107 160 197 105 247 0 0 0 0 0 0 0 0 0 1 255 1]
>> forWrite: false
>> peekBuffer: nil
>> pharo> show StdioStream>>#atEnd
>> StdioStream>>#atEnd
>> atEnd
>> 
>>^ file atEnd
>> pharo> get process.list
>> Morphic UI Process
>> Delay Scheduling Process
>> Low Space Watcher
>> Input Event Fetcher Process
>> Idle Process
>> WeakArray Finalization Process
>> CommandLine handler process
>> pharo> quit
>> Bye!
>> a NeoConsoleStdio
>> 
>> I know there are many approaches to a REPL, I don't claim mine is best, it 
>> is just the one that I have been using for years.
>> 
>> In the above, I do not depend on EOF - just to be clear. The point being 
>> that there is no immediate fundamental problem.
>> 
>> But there is something wrong with what is returned by Stdio stdin
>> 
>> Sven
>> 
>> 
>> 
> 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Stephane Ducasse]

I did not know about the NeoConsole. Nice because I wanted to build a
little REPL for my minilanguage implementation.

Stef

On Wed, Apr 11, 2018 at 8:47 PM, Sven Van Caekenberghe  wrote:
> Alistair,
>
>> On 11 Apr 2018, at 19:42, Sven Van Caekenberghe  wrote:
>>
>> I will send you some code later on.
>
> Today I arranged for my NeoConsole code (that normally works over a network 
> connection) to work over stdio. Although I am not yet happy with every aspect 
> of the implementation, it does work (using unaltered Zn streams and code). 
> The foll
>
> $ cat /etc/issue
> Ubuntu 16.04.4 LTS
> $ mkdir pharo7
> $ cd pharo7/
> $ curl get.pharo.org/70+vm | bash
>   % Total% Received % Xferd  Average Speed   TimeTime Time  
> Current
>  Dload  Upload   Total   SpentLeft  Speed
> 100  3036  100  30360 0  36799  0 --:--:-- --:--:-- --:--:-- 37024
> Downloading the latest 70 Image:
> http://files.pharo.org/get-files/70/pharo.zip
> Pharo.image
> Downloading the latest pharoVM:
> http://files.pharo.org/get-files/70/pharo-linux-stable.zip
> pharo-vm/pharo
> Creating starter scripts pharo and pharo-ui
> On a 64-bit system? You must enable and install the 32-bit libraries
>Please see http://pharo.org/gnu-linux-installation for detailed 
> instructions
> $ ./pharo Pharo.image config http://mc.stfx.eu/Neo ConfigurationOfNeoConsole 
> --install=bleedingEdge
> 'Installing ConfigurationOfNeoConsole bleedingEdge'
>
> Loading 1-baseline of ConfigurationOfNeoConsole...
> Fetched -> Neo-Console-Core-SvenVanCaekenberghe.24 --- http://mc.stfx.eu/Neo 
> --- http://mc.stfx.eu/Neo
> Loaded -> Neo-Console-Core-SvenVanCaekenberghe.24 --- http://mc.stfx.eu/Neo 
> --- cache
> ...finished 1-baseline
> $ ./pharo Pharo.image eval NeoConsoleStdio run
> Neo Console 
> Pharo-7.0+alpha.build.760.sha.d2734dcabda799803c307365bcd120f92211d34a (32 
> Bit)
> pharo> 1+2
>
> 3
> pharo> 42 factorial
>
> 14050061177528798985431426062445115699363840
> pharo> Stdio stdin
>
> StdioStream: #stdin
> pharo> ==
> self: StdioStream: #stdin
> class: StdioStream
> file: a File
> handle: #[148 213 25 107 160 197 105 247 0 0 0 0 0 0 0 0 0 1 255 1]
> forWrite: false
> peekBuffer: nil
> pharo> show StdioStream>>#atEnd
> StdioStream>>#atEnd
> atEnd
>
> ^ file atEnd
> pharo> get process.list
> Morphic UI Process
> Delay Scheduling Process
> Low Space Watcher
> Input Event Fetcher Process
> Idle Process
> WeakArray Finalization Process
> CommandLine handler process
> pharo> quit
> Bye!
> a NeoConsoleStdio
>
> I know there are many approaches to a REPL, I don't claim mine is best, it is 
> just the one that I have been using for years.
>
> In the above, I do not depend on EOF - just to be clear. The point being that 
> there is no immediate fundamental problem.
>
> But there is something wrong with what is returned by Stdio stdin
>
> Sven
>
>
>

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Stephane Ducasse]

On Wed, Apr 11, 2018 at 5:33 PM, Sven Van Caekenberghe  wrote:
> I think we have to reset this whole discussion.
>
>   FileStream stdin
>
> and
>
>   Stdio stdin
>
> are completely different !

Could you explain the difference because I'm dead tired and got confused :)

>
> We'll have to check that first, before talking about the issues raised in 
> this thread.
>
> And BTW these terminal streams are a real pain to test ;-)
>
>> On 11 Apr 2018, at 17:20, Sven Van Caekenberghe  wrote:
>>
>>
>>
>>> On 11 Apr 2018, at 17:16, Denis Kudriashov  wrote:
>>>
>>>
>>> 2018-04-11 17:02 GMT+02:00 Sven Van Caekenberghe :
>>>
>>>
 On 11 Apr 2018, at 16:36, Sven Van Caekenberghe  wrote:

 I can make your example, using the Zn variants, work with the following 
 change:

 StdioStream>>#atEnd
 ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ]
>>>
>>> Argh, make that
>>>
>>> atEnd
>>>  ^ peekBuffer isNil and: [ (peekBuffer := self next) isNil ]
>>>
>>> But discussion exactly about "self next isNil": how to avoid it.
>>
>> I know, but like this it could/might become an implementation detail.
>>
>> The more things that I try, the more that I feel that stdin is so special 
>> that it does not fit in the rest of the stream zoo.
>>
>>> but I am still testing, this is probably not the final answer/solution.
>>>
 Which is a literal implementation of your statement that you can only know 
 that you are atEnd by reading (and thus waiting/blocking) and checking for 
 nil, which seems logical to me, given the fact that you *are* waiting for 
 user input.

 BTW, at least on macOS you have to enter ctrl-D (^D) on a separate line, I 
 am not sure how relevant that is, but that is probably another argument 
 that stdin is special (being line-buffered by the OS, EOF needing to be on 
 a separate line).

 And FWIW, I have been writing networking code in Pharo for years, and I 
 have never had issues with unclear semantics of these primitives (#atEnd, 
 #next, #peek) on network streams, either the classic SocketStream or the 
 Zdc* streams (TLS or not). That is why I think we have to be careful.

 That being said, it is important to continue this discussion, I find it 
 very interesting. I am trying to write some test code using stdin myself, 
 to better understand the topic.

> On 11 Apr 2018, at 16:06, Alistair Grant  wrote:
>
> On 11 April 2018 at 15:11, Sven Van Caekenberghe  wrote:
>>
>>
>>> On 11 Apr 2018, at 11:12, Sven Van Caekenberghe  wrote:
>>>
>>> How does one modify #atEnd to block ? I suppose you are talking about 
>>> StdioStream>>#atEnd ?
>>>
>>> ^ self peek isNil
>>>
>>> ?
>>
>> Still the same question, how do you implement a blocking #atEnd for 
>> stdin ?
>>
>> I have seen your stdio.cs which is indeed needed as the current 
>> StdioStream>>#atEnd is bogus for sure.
>>
>> But that is still a non-blocking one, right ?
>>
>> Since there is a peekBuffer in StdioStream, why can't that be used ?
>
> I think you've created a chicken-and-egg problem with this question,
> but ignoring that for now:
>
>
> StdioStream>>peek
> "Answer what would be returned if the message next were sent to the
> receiver. If the receiver is at the end, answer nil.  "
>
>  self atEnd ifTrue: [^ nil ].
>
>  peekBuffer ifNotNil: [ ^ peekBuffer ].
>
>  ^ peekBuffer := self next.
>
>
>
> So when we first start the program, i.e. the user hasn't entered any
> input yet, and #peek is called:
>
> 1. #atEnd returns false because Ctrl-D (or similar) hasn't been
> entered (assuming it is non-blocking).
> 2. peekBuffer is nil because we haven't previously called #peek.
> 3. The system now blocks on "self next".
>
>
> Just a reminder: for terminal input the end-of-file isn't reached
> until the user explicitly enters the end of file key (Ctrl-D).
>
> So, if there is no buffered input (either none has been entered yet,
> or all input has been consumed)
>
> #atEnd (after the patch) calls #primAtEnd:.
>
> At the moment, #primAtEnd: ends up calling the libc function feof(),
> which is non-blocking and answers the end-of-file flag for the FILE*.
> Since the user hasn't entered Ctrl-D, that's false.
>
> If we want to control iteration over the stream and ensure that we
> don't need to do a "stream next == nil" check, then #primAtEnd: is
> going to have to peek for the next character, and that means waiting
> for the user to enter that character.
>
> In c that is typically done using:
>
> atEnd = ungetc(fgetc(fp), fp);
>
> and fgetc() will block until 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Stephane Ducasse]

On Wed, Apr 11, 2018 at 12:04 PM, Denis Kudriashov  wrote:
> Hi Alistair.
>
> I don't think anybody is annoyed by you. You are doing really good job. And
> nice thing that you are super patient to continue :)


+10

>
> What I try to understand is why blocking atEnd is bad?
> Here is code from VMMaker:
>
> [stdin atEnd] whileFalse:
> [| nextChunk |
> stdout nextPutAll: 'squeak> '; flush.
> nextChunk := stdin nextChunkNoTag.
> [nextChunk notEmpty and: [nextChunk first isSeparator]] whileTrue:
> [nextChunk := nextChunk allButFirst].
> Transcript cr; nextPutAll: nextChunk; cr; flush.
> [stdout print: (Compiler evaluate: nextChunk); cr; flush]
> on: Error
> do: [:ex| self logError: ex description inContext: ex signalerContext to:
> stderr]].
> quitOnEof ifTrue:
> [SourceFiles at: 2 put: nil.
> Smalltalk snapshot: false andQuit: true]
>
>
> I am not see why it breaks with blocking #atEnd. Can you explain?
>
> 2018-04-11 11:41 GMT+02:00 Alistair Grant :
>>
>> Hi Sven,
>>
>> Oh dear.  I feel as though I'm not getting my concerns across at all
>> well, and I'm pushing hard enough that all I'm going to do is make
>> people annoyed.  So let me try to restate the issue one last time
>> before answering your questions directly.
>>
>> Pharo & Squeak have unwritten rules about stream usage that I suspect
>> have just emerged over time without being designed.
>>
>> If you want to be able to iterate over any stream, and in particular
>> stdin from a terminal (which, as far as I know, is the outlier that
>> causes all the problems) you have to follow these rules:
>>
>> 1.  If the stream is character / byte oriented you have to check for
>> EOF using "stream next == nil".  #atEnd can be used, but you'll still
>> have to do the nil check.
>>
>> 2.  All other streams have to check for EOF (end of stream) using
>> #atEnd.  "stream next == nil" can be used, but you'll still need to
>> test #atEnd to determine whether nil is a value returned by the
>> stream.
>>
>> If you write code that you want to be able to consume characters,
>> bytes or any other object, you'll have to test both "stream next ==
>> nil" and #atEnd.
>>
>> The rules are the result of the original blue book design being that
>> #atEnd should be used, and then character input from a terminal being
>> added later, but always returning an EOF character (nil) before #atEnd
>> answers correctly.
>>
>> At the moment, ZnCharacterEncoder uses #atEnd on character / byte
>> streams, so fails for stdin on a terminal.
>>
>> Back to your questions:
>>
>> On 11 April 2018 at 11:12, Sven Van Caekenberghe  wrote:
>> >
>> >
>> >> On 11 Apr 2018, at 10:29, Alistair Grant  wrote:
>> >>
>> >> Hi Denis,
>> >>
>> >> On 11 April 2018 at 10:02, Denis Kudriashov 
>> >> wrote:
>> >>>
>> >>> 2018-04-11 8:32 GMT+02:00 Alistair Grant :
>> 
>> >>> Where is it being said that #next and/or #atEnd should be blocking
>> >>> or
>> >>> non-blocking ?
>> >>
>> >> There is existing code that assumes that #atEnd is non-blocking and
>> >> that #next is allowed block.  I believe that we should keep those
>> >> conditions.
>> >
>> > I fail to see where that is written down, either way. Can you point
>> > me
>> > to comments stating that, I would really like to know ?
>> 
>>  I'm not aware of it being written down, just that ever existing
>>  implementation I'm aware of behaves this way.
>> 
>>  On the other hand, making #atEnd blocking breaks Eliot's REPL sample
>>  (in Squeak).
>> >>>
>> >>>
>> >>> Could you write here this example, please?
>> >>
>> >> The code is loaded in squeak using:
>> >>
>> >>
>> >> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/buildspurtrunkreaderimage.sh
>> >>
>> >> for 32 bit images.  It loads:
>> >>
>> >>
>> >> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/LoadReader.st
>> >>
>> >> which loads package CogTools-Listener in
>> >> http://source.squeak.org/VMMaker
>> >>
>> >> An image that automatically runs the code and nothing else is created
>> >> in:
>> >>
>> >>
>> >> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/StartReader.st
>> >>
>> >>
>> >> If you want to run it interactively you can load CogTools-Listener and
>> >> do something like:
>> >>
>> >> StdioListener new
>> >>quitOnEof: false;
>> >>run
>> >
>> > What does #quitOnEof: do ? Can the StdioListener code be browsed/viewed
>> > online somewhere ?
>>
>> I just referenced this as an example of making #atEnd (really
>> FilePlugin>>primitiveFileAtEnd) blocking causing problems.  I wasn't
>> expecting people to go and look at the code or use it as a test.
>>
>> If you really want to look at it (from Pharo):
>>
>> 1. Add http://source.squeak.org/VMMaker as a repository.
>> 2. Browse the CogTools-Listener package
>>
>>
>> >> If you modify #atEnd to block it 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

Alistair,

> On 11 Apr 2018, at 19:42, Sven Van Caekenberghe  wrote:
> 
> I will send you some code later on.

Today I arranged for my NeoConsole code (that normally works over a network 
connection) to work over stdio. Although I am not yet happy with every aspect 
of the implementation, it does work (using unaltered Zn streams and code). The 
foll

$ cat /etc/issue
Ubuntu 16.04.4 LTS
$ mkdir pharo7
$ cd pharo7/
$ curl get.pharo.org/70+vm | bash
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
100  3036  100  30360 0  36799  0 --:--:-- --:--:-- --:--:-- 37024
Downloading the latest 70 Image:
http://files.pharo.org/get-files/70/pharo.zip
Pharo.image
Downloading the latest pharoVM:
http://files.pharo.org/get-files/70/pharo-linux-stable.zip
pharo-vm/pharo
Creating starter scripts pharo and pharo-ui
On a 64-bit system? You must enable and install the 32-bit libraries
   Please see http://pharo.org/gnu-linux-installation for detailed instructions
$ ./pharo Pharo.image config http://mc.stfx.eu/Neo ConfigurationOfNeoConsole 
--install=bleedingEdge
'Installing ConfigurationOfNeoConsole bleedingEdge'

Loading 1-baseline of ConfigurationOfNeoConsole...
Fetched -> Neo-Console-Core-SvenVanCaekenberghe.24 --- http://mc.stfx.eu/Neo 
--- http://mc.stfx.eu/Neo
Loaded -> Neo-Console-Core-SvenVanCaekenberghe.24 --- http://mc.stfx.eu/Neo --- 
cache
...finished 1-baseline
$ ./pharo Pharo.image eval NeoConsoleStdio run
Neo Console 
Pharo-7.0+alpha.build.760.sha.d2734dcabda799803c307365bcd120f92211d34a (32 Bit)
pharo> 1+2

3
pharo> 42 factorial

14050061177528798985431426062445115699363840
pharo> Stdio stdin

StdioStream: #stdin
pharo> ==
self: StdioStream: #stdin
class: StdioStream
file: a File
handle: #[148 213 25 107 160 197 105 247 0 0 0 0 0 0 0 0 0 1 255 1]
forWrite: false
peekBuffer: nil
pharo> show StdioStream>>#atEnd
StdioStream>>#atEnd
atEnd

^ file atEnd
pharo> get process.list
Morphic UI Process
Delay Scheduling Process
Low Space Watcher
Input Event Fetcher Process
Idle Process
WeakArray Finalization Process
CommandLine handler process
pharo> quit
Bye!
a NeoConsoleStdio

I know there are many approaches to a REPL, I don't claim mine is best, it is 
just the one that I have been using for years.

In the above, I do not depend on EOF - just to be clear. The point being that 
there is no immediate fundamental problem.

But there is something wrong with what is returned by Stdio stdin

Sven

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Nicolas Cellier]

2018-04-11 19:08 GMT+02:00 Alistair Grant :

> Hi Sven,
>
> On 11 April 2018 at 18:53, Sven Van Caekenberghe  wrote:
> > Something is off (and/or I am getting crazy, probably both).
> >
> > $ ./pharo --headless Pharo.image eval '(FileStream stdin binary; next:
> 3)'
> > 123
> > #[49]
> >
> > ??
> >
> > This should return #[49 50 51] AFAIK.
>
> I haven't checked properly, but I can take a guess at this...
>
> The file read primitive in FilePlugin tries to read from stdin 1
> character at a time (presumably to get around the line buffering that
> is done, but I haven't gone through the history.  At one time I think
> it modified the terminal settings, but that code is #if'd out).  For
> all other streams it attempts to read the requested number of
> characters.
>
> StandardFileStream>>readInto:startingAt:count: assumes that
> primitiveFileRead will always attempt to read count bytes, but it
> actually only attempts to read 1.
>
> Cheers,
> Alistair
>
>
https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/150

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

Hmm, but why does the following work then ?

$ ./pharo --headless Pharo.image eval '(FileStream stdin next: 3)'
123
'123'

The binary case is an error though, for sure.

And Zn character decoding works on binary streams, while the buffering streams 
read in blocks.

And it always comes back to 'stdin is handled specially', is that so in C, 
generally speaking ?

I am still digging though. I will send you some code later on.

> On 11 Apr 2018, at 19:08, Alistair Grant  wrote:
> 
> Hi Sven,
> 
> On 11 April 2018 at 18:53, Sven Van Caekenberghe  wrote:
>> Something is off (and/or I am getting crazy, probably both).
>> 
>> $ ./pharo --headless Pharo.image eval '(FileStream stdin binary; next: 3)'
>> 123
>> #[49]
>> 
>> ??
>> 
>> This should return #[49 50 51] AFAIK.
> 
> I haven't checked properly, but I can take a guess at this...
> 
> The file read primitive in FilePlugin tries to read from stdin 1
> character at a time (presumably to get around the line buffering that
> is done, but I haven't gone through the history.  At one time I think
> it modified the terminal settings, but that code is #if'd out).  For
> all other streams it attempts to read the requested number of
> characters.
> 
> StandardFileStream>>readInto:startingAt:count: assumes that
> primitiveFileRead will always attempt to read count bytes, but it
> actually only attempts to read 1.
> 
> Cheers,
> Alistair
> 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

Hi Sven,

On 11 April 2018 at 18:53, Sven Van Caekenberghe  wrote:
> Something is off (and/or I am getting crazy, probably both).
>
> $ ./pharo --headless Pharo.image eval '(FileStream stdin binary; next: 3)'
> 123
> #[49]
>
> ??
>
> This should return #[49 50 51] AFAIK.

I haven't checked properly, but I can take a guess at this...

The file read primitive in FilePlugin tries to read from stdin 1
character at a time (presumably to get around the line buffering that
is done, but I haven't gone through the history.  At one time I think
it modified the terminal settings, but that code is #if'd out).  For
all other streams it attempts to read the requested number of
characters.

StandardFileStream>>readInto:startingAt:count: assumes that
primitiveFileRead will always attempt to read count bytes, but it
actually only attempts to read 1.

Cheers,
Alistair

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

Something is off (and/or I am getting crazy, probably both).

$ ./pharo --headless Pharo.image eval '(FileStream stdin binary; next: 3)'
123
#[49]

??

This should return #[49 50 51] AFAIK.

> On 11 Apr 2018, at 18:26, Alistair Grant  wrote:
> 
> Hi Sven,
> 
> On 11 April 2018 at 17:33, Sven Van Caekenberghe  wrote:
>> I think we have to reset this whole discussion.
>> 
>>  FileStream stdin
>> 
>> and
>> 
>>  Stdio stdin
>> 
>> are completely different !
>> 
>> We'll have to check that first, before talking about the issues raised in 
>> this thread.
> 
> Are you sure you're comparing (roughly) equal things?
> 
> I would compare:
> 
> FileStream stdin
> 
> (very roughly) to:
> 
> ZnCharacterReadStream on:
>(ZnBufferedReadStream on:
>Stdio stdin).
> 
> Actually it is still not a fair comparison because MultiByteFileStream
> attempts to be writable as well.  If you could notionally do:
> 
> FileStream stdin
>  readOnly;
>  binary;
>  unbuffered;
>  yourself.
> 
> you could compare FileStream and Stdio :-)
> 
> One important similarity:  At the bottom they both use the same set of
> primitives to communicate with the OS stdio streams (FilePlugin).
> 
> HTH,
> Alistair
> 
> 
> 
>> And BTW these terminal streams are a real pain to test ;-)

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

Hi Sven,

On 11 April 2018 at 17:33, Sven Van Caekenberghe  wrote:
> I think we have to reset this whole discussion.
>
>   FileStream stdin
>
> and
>
>   Stdio stdin
>
> are completely different !
>
> We'll have to check that first, before talking about the issues raised in 
> this thread.

Are you sure you're comparing (roughly) equal things?

I would compare:

FileStream stdin

(very roughly) to:

ZnCharacterReadStream on:
(ZnBufferedReadStream on:
Stdio stdin).

Actually it is still not a fair comparison because MultiByteFileStream
attempts to be writable as well.  If you could notionally do:

FileStream stdin
  readOnly;
  binary;
  unbuffered;
  yourself.

you could compare FileStream and Stdio :-)

One important similarity:  At the bottom they both use the same set of
primitives to communicate with the OS stdio streams (FilePlugin).

HTH,
Alistair



> And BTW these terminal streams are a real pain to test ;-)

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

Hi Denis,

On 11 April 2018 at 17:16, Denis Kudriashov  wrote:
>
> 2018-04-11 17:02 GMT+02:00 Sven Van Caekenberghe :
>>
>>
>>
>> > On 11 Apr 2018, at 16:36, Sven Van Caekenberghe  wrote:
>> >
>> > I can make your example, using the Zn variants, work with the following
>> > change:
>> >
>> > StdioStream>>#atEnd
>> >  ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ]
>>
>> Argh, make that
>>
>> atEnd
>>   ^ peekBuffer isNil and: [ (peekBuffer := self next) isNil ]
>
>
> But discussion exactly about "self next isNil": how to avoid it.

Apologies in advance for being pedantic, but...

Do you really mean this (that the discussion is about how to avoid
testing "self next isNil")?

My argument has been that without making #atEnd blocking it is not
possible to avoid the test (and we don't want to make #atEnd
blocking).  All the existing stream code that deals with character /
byte streams does this test (see my "unwritten rules" from a previous
reply).  We don't want to make #atEnd blocking, so we need to keep the
test (and my personal opinion is that changing the Zinc streams to
adopt this approach does not add any significant architectural
complexity).

Assuming we reach agreement on the above, we do need to make the
"unwritten rules" written. If we reach a different agreement we should
document that.


Cheers,
Alistair

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

I think we have to reset this whole discussion.

  FileStream stdin 

and

  Stdio stdin

are completely different !

We'll have to check that first, before talking about the issues raised in this 
thread.

And BTW these terminal streams are a real pain to test ;-)

> On 11 Apr 2018, at 17:20, Sven Van Caekenberghe  wrote:
> 
> 
> 
>> On 11 Apr 2018, at 17:16, Denis Kudriashov  wrote:
>> 
>> 
>> 2018-04-11 17:02 GMT+02:00 Sven Van Caekenberghe :
>> 
>> 
>>> On 11 Apr 2018, at 16:36, Sven Van Caekenberghe  wrote:
>>> 
>>> I can make your example, using the Zn variants, work with the following 
>>> change:
>>> 
>>> StdioStream>>#atEnd
>>> ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ]
>> 
>> Argh, make that
>> 
>> atEnd
>>  ^ peekBuffer isNil and: [ (peekBuffer := self next) isNil ]
>> 
>> But discussion exactly about "self next isNil": how to avoid it.
> 
> I know, but like this it could/might become an implementation detail.
> 
> The more things that I try, the more that I feel that stdin is so special 
> that it does not fit in the rest of the stream zoo.
> 
>> but I am still testing, this is probably not the final answer/solution.
>> 
>>> Which is a literal implementation of your statement that you can only know 
>>> that you are atEnd by reading (and thus waiting/blocking) and checking for 
>>> nil, which seems logical to me, given the fact that you *are* waiting for 
>>> user input.
>>> 
>>> BTW, at least on macOS you have to enter ctrl-D (^D) on a separate line, I 
>>> am not sure how relevant that is, but that is probably another argument 
>>> that stdin is special (being line-buffered by the OS, EOF needing to be on 
>>> a separate line).
>>> 
>>> And FWIW, I have been writing networking code in Pharo for years, and I 
>>> have never had issues with unclear semantics of these primitives (#atEnd, 
>>> #next, #peek) on network streams, either the classic SocketStream or the 
>>> Zdc* streams (TLS or not). That is why I think we have to be careful.
>>> 
>>> That being said, it is important to continue this discussion, I find it 
>>> very interesting. I am trying to write some test code using stdin myself, 
>>> to better understand the topic.
>>> 
 On 11 Apr 2018, at 16:06, Alistair Grant  wrote:
 
 On 11 April 2018 at 15:11, Sven Van Caekenberghe  wrote:
> 
> 
>> On 11 Apr 2018, at 11:12, Sven Van Caekenberghe  wrote:
>> 
>> How does one modify #atEnd to block ? I suppose you are talking about 
>> StdioStream>>#atEnd ?
>> 
>> ^ self peek isNil
>> 
>> ?
> 
> Still the same question, how do you implement a blocking #atEnd for stdin 
> ?
> 
> I have seen your stdio.cs which is indeed needed as the current 
> StdioStream>>#atEnd is bogus for sure.
> 
> But that is still a non-blocking one, right ?
> 
> Since there is a peekBuffer in StdioStream, why can't that be used ?
 
 I think you've created a chicken-and-egg problem with this question,
 but ignoring that for now:
 
 
 StdioStream>>peek
 "Answer what would be returned if the message next were sent to the
 receiver. If the receiver is at the end, answer nil.  "
 
  self atEnd ifTrue: [^ nil ].
 
  peekBuffer ifNotNil: [ ^ peekBuffer ].
 
  ^ peekBuffer := self next.
 
 
 
 So when we first start the program, i.e. the user hasn't entered any
 input yet, and #peek is called:
 
 1. #atEnd returns false because Ctrl-D (or similar) hasn't been
 entered (assuming it is non-blocking).
 2. peekBuffer is nil because we haven't previously called #peek.
 3. The system now blocks on "self next".
 
 
 Just a reminder: for terminal input the end-of-file isn't reached
 until the user explicitly enters the end of file key (Ctrl-D).
 
 So, if there is no buffered input (either none has been entered yet,
 or all input has been consumed)
 
 #atEnd (after the patch) calls #primAtEnd:.
 
 At the moment, #primAtEnd: ends up calling the libc function feof(),
 which is non-blocking and answers the end-of-file flag for the FILE*.
 Since the user hasn't entered Ctrl-D, that's false.
 
 If we want to control iteration over the stream and ensure that we
 don't need to do a "stream next == nil" check, then #primAtEnd: is
 going to have to peek for the next character, and that means waiting
 for the user to enter that character.
 
 In c that is typically done using:
 
 atEnd = ungetc(fgetc(fp), fp);
 
 and fgetc() will block until the user enters something.
 
> I have run your example testAtEnd.st now, and it works/fails as 
> advertised.
 
 :-)
 
 
 Cheers,
 Alistair

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

> On 11 Apr 2018, at 17:16, Denis Kudriashov  wrote:
> 
> 
> 2018-04-11 17:02 GMT+02:00 Sven Van Caekenberghe :
> 
> 
> > On 11 Apr 2018, at 16:36, Sven Van Caekenberghe  wrote:
> >
> > I can make your example, using the Zn variants, work with the following 
> > change:
> >
> > StdioStream>>#atEnd
> >  ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ]
> 
> Argh, make that
> 
> atEnd
>   ^ peekBuffer isNil and: [ (peekBuffer := self next) isNil ]
> 
> But discussion exactly about "self next isNil": how to avoid it.

I know, but like this it could/might become an implementation detail.

The more things that I try, the more that I feel that stdin is so special that 
it does not fit in the rest of the stream zoo.

> but I am still testing, this is probably not the final answer/solution.
> 
> > Which is a literal implementation of your statement that you can only know 
> > that you are atEnd by reading (and thus waiting/blocking) and checking for 
> > nil, which seems logical to me, given the fact that you *are* waiting for 
> > user input.
> >
> > BTW, at least on macOS you have to enter ctrl-D (^D) on a separate line, I 
> > am not sure how relevant that is, but that is probably another argument 
> > that stdin is special (being line-buffered by the OS, EOF needing to be on 
> > a separate line).
> >
> > And FWIW, I have been writing networking code in Pharo for years, and I 
> > have never had issues with unclear semantics of these primitives (#atEnd, 
> > #next, #peek) on network streams, either the classic SocketStream or the 
> > Zdc* streams (TLS or not). That is why I think we have to be careful.
> >
> > That being said, it is important to continue this discussion, I find it 
> > very interesting. I am trying to write some test code using stdin myself, 
> > to better understand the topic.
> >
> >> On 11 Apr 2018, at 16:06, Alistair Grant  wrote:
> >>
> >> On 11 April 2018 at 15:11, Sven Van Caekenberghe  wrote:
> >>>
> >>>
>  On 11 Apr 2018, at 11:12, Sven Van Caekenberghe  wrote:
> 
>  How does one modify #atEnd to block ? I suppose you are talking about 
>  StdioStream>>#atEnd ?
> 
>  ^ self peek isNil
> 
>  ?
> >>>
> >>> Still the same question, how do you implement a blocking #atEnd for stdin 
> >>> ?
> >>>
> >>> I have seen your stdio.cs which is indeed needed as the current 
> >>> StdioStream>>#atEnd is bogus for sure.
> >>>
> >>> But that is still a non-blocking one, right ?
> >>>
> >>> Since there is a peekBuffer in StdioStream, why can't that be used ?
> >>
> >> I think you've created a chicken-and-egg problem with this question,
> >> but ignoring that for now:
> >>
> >>
> >> StdioStream>>peek
> >> "Answer what would be returned if the message next were sent to the
> >> receiver. If the receiver is at the end, answer nil.  "
> >>
> >>   self atEnd ifTrue: [^ nil ].
> >>
> >>   peekBuffer ifNotNil: [ ^ peekBuffer ].
> >>
> >>   ^ peekBuffer := self next.
> >>
> >>
> >>
> >> So when we first start the program, i.e. the user hasn't entered any
> >> input yet, and #peek is called:
> >>
> >> 1. #atEnd returns false because Ctrl-D (or similar) hasn't been
> >> entered (assuming it is non-blocking).
> >> 2. peekBuffer is nil because we haven't previously called #peek.
> >> 3. The system now blocks on "self next".
> >>
> >>
> >> Just a reminder: for terminal input the end-of-file isn't reached
> >> until the user explicitly enters the end of file key (Ctrl-D).
> >>
> >> So, if there is no buffered input (either none has been entered yet,
> >> or all input has been consumed)
> >>
> >> #atEnd (after the patch) calls #primAtEnd:.
> >>
> >> At the moment, #primAtEnd: ends up calling the libc function feof(),
> >> which is non-blocking and answers the end-of-file flag for the FILE*.
> >> Since the user hasn't entered Ctrl-D, that's false.
> >>
> >> If we want to control iteration over the stream and ensure that we
> >> don't need to do a "stream next == nil" check, then #primAtEnd: is
> >> going to have to peek for the next character, and that means waiting
> >> for the user to enter that character.
> >>
> >> In c that is typically done using:
> >>
> >> atEnd = ungetc(fgetc(fp), fp);
> >>
> >> and fgetc() will block until the user enters something.
> >>
> >>> I have run your example testAtEnd.st now, and it works/fails as 
> >>> advertised.
> >>
> >> :-)
> >>
> >>
> >> Cheers,
> >> Alistair
> >

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Denis Kudriashov]

2018-04-11 17:02 GMT+02:00 Sven Van Caekenberghe :

>
>
> > On 11 Apr 2018, at 16:36, Sven Van Caekenberghe  wrote:
> >
> > I can make your example, using the Zn variants, work with the following
> change:
> >
> > StdioStream>>#atEnd
> >  ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ]
>
> Argh, make that
>
> atEnd
>   ^ peekBuffer isNil and: [ (peekBuffer := self next) isNil ]
>

But discussion exactly about "self next isNil": how to avoid it.


>
> but I am still testing, this is probably not the final answer/solution.
>
> > Which is a literal implementation of your statement that you can only
> know that you are atEnd by reading (and thus waiting/blocking) and checking
> for nil, which seems logical to me, given the fact that you *are* waiting
> for user input.
> >
> > BTW, at least on macOS you have to enter ctrl-D (^D) on a separate line,
> I am not sure how relevant that is, but that is probably another argument
> that stdin is special (being line-buffered by the OS, EOF needing to be on
> a separate line).
> >
> > And FWIW, I have been writing networking code in Pharo for years, and I
> have never had issues with unclear semantics of these primitives (#atEnd,
> #next, #peek) on network streams, either the classic SocketStream or the
> Zdc* streams (TLS or not). That is why I think we have to be careful.
> >
> > That being said, it is important to continue this discussion, I find it
> very interesting. I am trying to write some test code using stdin myself,
> to better understand the topic.
> >
> >> On 11 Apr 2018, at 16:06, Alistair Grant  wrote:
> >>
> >> On 11 April 2018 at 15:11, Sven Van Caekenberghe  wrote:
> >>>
> >>>
>  On 11 Apr 2018, at 11:12, Sven Van Caekenberghe  wrote:
> 
>  How does one modify #atEnd to block ? I suppose you are talking about
> StdioStream>>#atEnd ?
> 
>  ^ self peek isNil
> 
>  ?
> >>>
> >>> Still the same question, how do you implement a blocking #atEnd for
> stdin ?
> >>>
> >>> I have seen your stdio.cs which is indeed needed as the current
> StdioStream>>#atEnd is bogus for sure.
> >>>
> >>> But that is still a non-blocking one, right ?
> >>>
> >>> Since there is a peekBuffer in StdioStream, why can't that be used ?
> >>
> >> I think you've created a chicken-and-egg problem with this question,
> >> but ignoring that for now:
> >>
> >>
> >> StdioStream>>peek
> >> "Answer what would be returned if the message next were sent to the
> >> receiver. If the receiver is at the end, answer nil.  "
> >>
> >>   self atEnd ifTrue: [^ nil ].
> >>
> >>   peekBuffer ifNotNil: [ ^ peekBuffer ].
> >>
> >>   ^ peekBuffer := self next.
> >>
> >>
> >>
> >> So when we first start the program, i.e. the user hasn't entered any
> >> input yet, and #peek is called:
> >>
> >> 1. #atEnd returns false because Ctrl-D (or similar) hasn't been
> >> entered (assuming it is non-blocking).
> >> 2. peekBuffer is nil because we haven't previously called #peek.
> >> 3. The system now blocks on "self next".
> >>
> >>
> >> Just a reminder: for terminal input the end-of-file isn't reached
> >> until the user explicitly enters the end of file key (Ctrl-D).
> >>
> >> So, if there is no buffered input (either none has been entered yet,
> >> or all input has been consumed)
> >>
> >> #atEnd (after the patch) calls #primAtEnd:.
> >>
> >> At the moment, #primAtEnd: ends up calling the libc function feof(),
> >> which is non-blocking and answers the end-of-file flag for the FILE*.
> >> Since the user hasn't entered Ctrl-D, that's false.
> >>
> >> If we want to control iteration over the stream and ensure that we
> >> don't need to do a "stream next == nil" check, then #primAtEnd: is
> >> going to have to peek for the next character, and that means waiting
> >> for the user to enter that character.
> >>
> >> In c that is typically done using:
> >>
> >> atEnd = ungetc(fgetc(fp), fp);
> >>
> >> and fgetc() will block until the user enters something.
> >>
> >>> I have run your example testAtEnd.st now, and it works/fails as
> advertised.
> >>
> >> :-)
> >>
> >>
> >> Cheers,
> >> Alistair
> >
>
>
>

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

> On 11 Apr 2018, at 16:36, Sven Van Caekenberghe  wrote:
> 
> I can make your example, using the Zn variants, work with the following 
> change:
> 
> StdioStream>>#atEnd
>  ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ]

Argh, make that

atEnd
  ^ peekBuffer isNil and: [ (peekBuffer := self next) isNil ]

but I am still testing, this is probably not the final answer/solution.

> Which is a literal implementation of your statement that you can only know 
> that you are atEnd by reading (and thus waiting/blocking) and checking for 
> nil, which seems logical to me, given the fact that you *are* waiting for 
> user input.
> 
> BTW, at least on macOS you have to enter ctrl-D (^D) on a separate line, I am 
> not sure how relevant that is, but that is probably another argument that 
> stdin is special (being line-buffered by the OS, EOF needing to be on a 
> separate line).
> 
> And FWIW, I have been writing networking code in Pharo for years, and I have 
> never had issues with unclear semantics of these primitives (#atEnd, #next, 
> #peek) on network streams, either the classic SocketStream or the Zdc* 
> streams (TLS or not). That is why I think we have to be careful.
> 
> That being said, it is important to continue this discussion, I find it very 
> interesting. I am trying to write some test code using stdin myself, to 
> better understand the topic.
> 
>> On 11 Apr 2018, at 16:06, Alistair Grant  wrote:
>> 
>> On 11 April 2018 at 15:11, Sven Van Caekenberghe  wrote:
>>> 
>>> 
 On 11 Apr 2018, at 11:12, Sven Van Caekenberghe  wrote:
 
 How does one modify #atEnd to block ? I suppose you are talking about 
 StdioStream>>#atEnd ?
 
 ^ self peek isNil
 
 ?
>>> 
>>> Still the same question, how do you implement a blocking #atEnd for stdin ?
>>> 
>>> I have seen your stdio.cs which is indeed needed as the current 
>>> StdioStream>>#atEnd is bogus for sure.
>>> 
>>> But that is still a non-blocking one, right ?
>>> 
>>> Since there is a peekBuffer in StdioStream, why can't that be used ?
>> 
>> I think you've created a chicken-and-egg problem with this question,
>> but ignoring that for now:
>> 
>> 
>> StdioStream>>peek
>> "Answer what would be returned if the message next were sent to the
>> receiver. If the receiver is at the end, answer nil.  "
>> 
>>   self atEnd ifTrue: [^ nil ].
>> 
>>   peekBuffer ifNotNil: [ ^ peekBuffer ].
>> 
>>   ^ peekBuffer := self next.
>> 
>> 
>> 
>> So when we first start the program, i.e. the user hasn't entered any
>> input yet, and #peek is called:
>> 
>> 1. #atEnd returns false because Ctrl-D (or similar) hasn't been
>> entered (assuming it is non-blocking).
>> 2. peekBuffer is nil because we haven't previously called #peek.
>> 3. The system now blocks on "self next".
>> 
>> 
>> Just a reminder: for terminal input the end-of-file isn't reached
>> until the user explicitly enters the end of file key (Ctrl-D).
>> 
>> So, if there is no buffered input (either none has been entered yet,
>> or all input has been consumed)
>> 
>> #atEnd (after the patch) calls #primAtEnd:.
>> 
>> At the moment, #primAtEnd: ends up calling the libc function feof(),
>> which is non-blocking and answers the end-of-file flag for the FILE*.
>> Since the user hasn't entered Ctrl-D, that's false.
>> 
>> If we want to control iteration over the stream and ensure that we
>> don't need to do a "stream next == nil" check, then #primAtEnd: is
>> going to have to peek for the next character, and that means waiting
>> for the user to enter that character.
>> 
>> In c that is typically done using:
>> 
>> atEnd = ungetc(fgetc(fp), fp);
>> 
>> and fgetc() will block until the user enters something.
>> 
>>> I have run your example testAtEnd.st now, and it works/fails as advertised.
>> 
>> :-)
>> 
>> 
>> Cheers,
>> Alistair
> 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

I can make your example, using the Zn variants, work with the following change:

StdioStream>>#atEnd
  ^ peekBuffer isNil or: [ (peekBuffer := self next) isNil ]

Which is a literal implementation of your statement that you can only know that 
you are atEnd by reading (and thus waiting/blocking) and checking for nil, 
which seems logical to me, given the fact that you *are* waiting for user input.

BTW, at least on macOS you have to enter ctrl-D (^D) on a separate line, I am 
not sure how relevant that is, but that is probably another argument that stdin 
is special (being line-buffered by the OS, EOF needing to be on a separate 
line).

And FWIW, I have been writing networking code in Pharo for years, and I have 
never had issues with unclear semantics of these primitives (#atEnd, #next, 
#peek) on network streams, either the classic SocketStream or the Zdc* streams 
(TLS or not). That is why I think we have to be careful.

That being said, it is important to continue this discussion, I find it very 
interesting. I am trying to write some test code using stdin myself, to better 
understand the topic.

> On 11 Apr 2018, at 16:06, Alistair Grant  wrote:
> 
> On 11 April 2018 at 15:11, Sven Van Caekenberghe  wrote:
>> 
>> 
>>> On 11 Apr 2018, at 11:12, Sven Van Caekenberghe  wrote:
>>> 
>>> How does one modify #atEnd to block ? I suppose you are talking about 
>>> StdioStream>>#atEnd ?
>>> 
>>> ^ self peek isNil
>>> 
>>> ?
>> 
>> Still the same question, how do you implement a blocking #atEnd for stdin ?
>> 
>> I have seen your stdio.cs which is indeed needed as the current 
>> StdioStream>>#atEnd is bogus for sure.
>> 
>> But that is still a non-blocking one, right ?
>> 
>> Since there is a peekBuffer in StdioStream, why can't that be used ?
> 
> I think you've created a chicken-and-egg problem with this question,
> but ignoring that for now:
> 
> 
> StdioStream>>peek
> "Answer what would be returned if the message next were sent to the
> receiver. If the receiver is at the end, answer nil.  "
> 
>self atEnd ifTrue: [^ nil ].
> 
>peekBuffer ifNotNil: [ ^ peekBuffer ].
> 
>^ peekBuffer := self next.
> 
> 
> 
> So when we first start the program, i.e. the user hasn't entered any
> input yet, and #peek is called:
> 
> 1. #atEnd returns false because Ctrl-D (or similar) hasn't been
> entered (assuming it is non-blocking).
> 2. peekBuffer is nil because we haven't previously called #peek.
> 3. The system now blocks on "self next".
> 
> 
> Just a reminder: for terminal input the end-of-file isn't reached
> until the user explicitly enters the end of file key (Ctrl-D).
> 
> So, if there is no buffered input (either none has been entered yet,
> or all input has been consumed)
> 
> #atEnd (after the patch) calls #primAtEnd:.
> 
> At the moment, #primAtEnd: ends up calling the libc function feof(),
> which is non-blocking and answers the end-of-file flag for the FILE*.
> Since the user hasn't entered Ctrl-D, that's false.
> 
> If we want to control iteration over the stream and ensure that we
> don't need to do a "stream next == nil" check, then #primAtEnd: is
> going to have to peek for the next character, and that means waiting
> for the user to enter that character.
> 
> In c that is typically done using:
> 
> atEnd = ungetc(fgetc(fp), fp);
> 
> and fgetc() will block until the user enters something.
> 
>> I have run your example testAtEnd.st now, and it works/fails as advertised.
> 
> :-)
> 
> 
> Cheers,
> Alistair

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

On 11 April 2018 at 15:11, Sven Van Caekenberghe  wrote:
>
>
>> On 11 Apr 2018, at 11:12, Sven Van Caekenberghe  wrote:
>>
>> How does one modify #atEnd to block ? I suppose you are talking about 
>> StdioStream>>#atEnd ?
>>
>> ^ self peek isNil
>>
>> ?
>
> Still the same question, how do you implement a blocking #atEnd for stdin ?
>
> I have seen your stdio.cs which is indeed needed as the current 
> StdioStream>>#atEnd is bogus for sure.
>
> But that is still a non-blocking one, right ?
>
> Since there is a peekBuffer in StdioStream, why can't that be used ?

I think you've created a chicken-and-egg problem with this question,
but ignoring that for now:


StdioStream>>peek
"Answer what would be returned if the message next were sent to the
receiver. If the receiver is at the end, answer nil.  "

self atEnd ifTrue: [^ nil ].

peekBuffer ifNotNil: [ ^ peekBuffer ].

^ peekBuffer := self next.



So when we first start the program, i.e. the user hasn't entered any
input yet, and #peek is called:

1. #atEnd returns false because Ctrl-D (or similar) hasn't been
entered (assuming it is non-blocking).
2. peekBuffer is nil because we haven't previously called #peek.
3. The system now blocks on "self next".


Just a reminder: for terminal input the end-of-file isn't reached
until the user explicitly enters the end of file key (Ctrl-D).

So, if there is no buffered input (either none has been entered yet,
or all input has been consumed)

#atEnd (after the patch) calls #primAtEnd:.

At the moment, #primAtEnd: ends up calling the libc function feof(),
which is non-blocking and answers the end-of-file flag for the FILE*.
Since the user hasn't entered Ctrl-D, that's false.

If we want to control iteration over the stream and ensure that we
don't need to do a "stream next == nil" check, then #primAtEnd: is
going to have to peek for the next character, and that means waiting
for the user to enter that character.

In c that is typically done using:

atEnd = ungetc(fgetc(fp), fp);

and fgetc() will block until the user enters something.

> I have run your example testAtEnd.st now, and it works/fails as advertised.

:-)


Cheers,
Alistair

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[David T. Lewis]

OS pipes are a similar case. On Pharo, you can run CommandShellTestCase to
provide some test coverage for this.

Dave

On Wed, Apr 11, 2018 at 03:13:35PM +0200, Denis Kudriashov wrote:
> Thanks for explanation.
> 
> I think it would be same scenario for socket stream where #atEnd is not
> blocking. So I agree that it is expected behaviour.
> 
> The example is general enough to expect it to be working for any given pair
> of in and out streams. So our streams should support this.
> 
> 
> 2018-04-11 14:56 GMT+02:00 Alistair Grant :
> 
> > Hi Sven & Dennis,
> >
> > On 11 April 2018 at 12:04, Denis Kudriashov  wrote:
> > > Hi Alistair.
> > >
> > > I don't think anybody is annoyed by you. You are doing really good job.
> > And
> > > nice thing that you are super patient to continue :)
> > >
> >
> > On 11 April 2018 at 12:13, Sven Van Caekenberghe  wrote:
> > >
> > > Yes, Alistair, you are a top notch open source contributor !
> > >
> > > For me, this discussion is about the difference between looking from low
> > level details/issues/changes up, vs, from a higher level down.
> >
> > Thanks for your kind words.
> >
> >
> >
> > > What I try to understand is why blocking atEnd is bad?
> > > Here is code from VMMaker:
> > >
> > > [stdin atEnd] whileFalse:
> > > [| nextChunk |
> > > stdout nextPutAll: 'squeak> '; flush.
> > > nextChunk := stdin nextChunkNoTag.
> > > [nextChunk notEmpty and: [nextChunk first isSeparator]] whileTrue:
> > > [nextChunk := nextChunk allButFirst].
> > > Transcript cr; nextPutAll: nextChunk; cr; flush.
> > > [stdout print: (Compiler evaluate: nextChunk); cr; flush]
> > > on: Error
> > > do: [:ex| self logError: ex description inContext: ex signalerContext to:
> > > stderr]].
> > > quitOnEof ifTrue:
> > > [SourceFiles at: 2 put: nil.
> > > Smalltalk snapshot: false andQuit: true]
> > >
> > >
> > > I am not see why it breaks with blocking #atEnd. Can you explain?
> >
> >
> > First consider the case where #atEnd doesn't block and we just want to
> > evaluate 4+3:
> >
> > 1. #atEnd will return false
> > 2. the loop will print the prompt
> > 3. wait for input (stdin nextChunkNoTag)
> > 4. print the result
> > 5. goto 1.
> >
> > So the screen will look like:
> >
> > squeak> 4+3!
> > 7
> > squeak> [cursor here]
> >
> > Which is what we expect (prompt, input, result, prompt).
> >
> > If #atEnd is blocking the VM will hang at step 1 until the user enters
> > something in the terminal.  In Ubuntu at least terminal input appears to
> > be line buffered, so for the example above the terminal will look like:
> >
> > 4+3!
> > squeak> 7
> > [cursor here]
> >
> > We don't get the prompt when the program is started, the result is
> > printed after the prompt, and then there's just a cursor sitting at the
> > start of the next line.
> >
> > Obviously the program could be re-written to have the correct output
> > with #atEnd blocking.  But I'm arguing that this program is
> > representative of many others, and we don't want to break backward
> > compatibility in this case.
> >
> > Cheers,
> > Alistair
> >
> >
> >
> >
> >
> >
> >
> >
> > > 2018-04-11 11:41 GMT+02:00 Alistair Grant :
> > >>
> > >> Hi Sven,
> > >>
> > >> Oh dear.  I feel as though I'm not getting my concerns across at all
> > >> well, and I'm pushing hard enough that all I'm going to do is make
> > >> people annoyed.  So let me try to restate the issue one last time
> > >> before answering your questions directly.
> > >>
> > >> Pharo & Squeak have unwritten rules about stream usage that I suspect
> > >> have just emerged over time without being designed.
> > >>
> > >> If you want to be able to iterate over any stream, and in particular
> > >> stdin from a terminal (which, as far as I know, is the outlier that
> > >> causes all the problems) you have to follow these rules:
> > >>
> > >> 1.  If the stream is character / byte oriented you have to check for
> > >> EOF using "stream next == nil".  #atEnd can be used, but you'll still
> > >> have to do the nil check.
> > >>
> > >> 2.  All other streams have to check for EOF (end of stream) using
> > >> #atEnd.  "stream next == nil" can be used, but you'll still need to
> > >> test #atEnd to determine whether nil is a value returned by the
> > >> stream.
> > >>
> > >> If you write code that you want to be able to consume characters,
> > >> bytes or any other object, you'll have to test both "stream next ==
> > >> nil" and #atEnd.
> > >>
> > >> The rules are the result of the original blue book design being that
> > >> #atEnd should be used, and then character input from a terminal being
> > >> added later, but always returning an EOF character (nil) before #atEnd
> > >> answers correctly.
> > >>
> > >> At the moment, ZnCharacterEncoder uses #atEnd on character / byte
> > >> streams, so fails for stdin on a terminal.
> > >>
> > >> Back to your questions:
> > >>
> > >> On 11 April 2018 at 11:12, Sven Van Caekenberghe 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Nicolas Cellier]

Hi Alistair,
I must take my part too: I suggested that we could use a pair of
getc/ungetc to know if we are atEnd(OfData), but this obviously works well
with AsyncFileIO only, else it blocks.
For files, this generally isn't a problem (but maybe for network mounted
partitions), the latency is bearable, but for sockets and pipes, that is
not the right thing.

2018-04-11 14:56 GMT+02:00 Alistair Grant :

> Hi Sven & Dennis,
>
> On 11 April 2018 at 12:04, Denis Kudriashov  wrote:
> > Hi Alistair.
> >
> > I don't think anybody is annoyed by you. You are doing really good job.
> And
> > nice thing that you are super patient to continue :)
> >
>
> On 11 April 2018 at 12:13, Sven Van Caekenberghe  wrote:
> >
> > Yes, Alistair, you are a top notch open source contributor !
> >
> > For me, this discussion is about the difference between looking from low
> level details/issues/changes up, vs, from a higher level down.
>
> Thanks for your kind words.
>
>
>
> > What I try to understand is why blocking atEnd is bad?
> > Here is code from VMMaker:
> >
> > [stdin atEnd] whileFalse:
> > [| nextChunk |
> > stdout nextPutAll: 'squeak> '; flush.
> > nextChunk := stdin nextChunkNoTag.
> > [nextChunk notEmpty and: [nextChunk first isSeparator]] whileTrue:
> > [nextChunk := nextChunk allButFirst].
> > Transcript cr; nextPutAll: nextChunk; cr; flush.
> > [stdout print: (Compiler evaluate: nextChunk); cr; flush]
> > on: Error
> > do: [:ex| self logError: ex description inContext: ex signalerContext to:
> > stderr]].
> > quitOnEof ifTrue:
> > [SourceFiles at: 2 put: nil.
> > Smalltalk snapshot: false andQuit: true]
> >
> >
> > I am not see why it breaks with blocking #atEnd. Can you explain?
>
>
> First consider the case where #atEnd doesn't block and we just want to
> evaluate 4+3:
>
> 1. #atEnd will return false
> 2. the loop will print the prompt
> 3. wait for input (stdin nextChunkNoTag)
> 4. print the result
> 5. goto 1.
>
> So the screen will look like:
>
> squeak> 4+3!
> 7
> squeak> [cursor here]
>
> Which is what we expect (prompt, input, result, prompt).
>
> If #atEnd is blocking the VM will hang at step 1 until the user enters
> something in the terminal.  In Ubuntu at least terminal input appears to
> be line buffered, so for the example above the terminal will look like:
>
> 4+3!
> squeak> 7
> [cursor here]
>
> We don't get the prompt when the program is started, the result is
> printed after the prompt, and then there's just a cursor sitting at the
> start of the next line.
>
> Obviously the program could be re-written to have the correct output
> with #atEnd blocking.  But I'm arguing that this program is
> representative of many others, and we don't want to break backward
> compatibility in this case.
>
> Cheers,
> Alistair
>
>
>
>
>
>
>
>
> > 2018-04-11 11:41 GMT+02:00 Alistair Grant :
> >>
> >> Hi Sven,
> >>
> >> Oh dear.  I feel as though I'm not getting my concerns across at all
> >> well, and I'm pushing hard enough that all I'm going to do is make
> >> people annoyed.  So let me try to restate the issue one last time
> >> before answering your questions directly.
> >>
> >> Pharo & Squeak have unwritten rules about stream usage that I suspect
> >> have just emerged over time without being designed.
> >>
> >> If you want to be able to iterate over any stream, and in particular
> >> stdin from a terminal (which, as far as I know, is the outlier that
> >> causes all the problems) you have to follow these rules:
> >>
> >> 1.  If the stream is character / byte oriented you have to check for
> >> EOF using "stream next == nil".  #atEnd can be used, but you'll still
> >> have to do the nil check.
> >>
> >> 2.  All other streams have to check for EOF (end of stream) using
> >> #atEnd.  "stream next == nil" can be used, but you'll still need to
> >> test #atEnd to determine whether nil is a value returned by the
> >> stream.
> >>
> >> If you write code that you want to be able to consume characters,
> >> bytes or any other object, you'll have to test both "stream next ==
> >> nil" and #atEnd.
> >>
> >> The rules are the result of the original blue book design being that
> >> #atEnd should be used, and then character input from a terminal being
> >> added later, but always returning an EOF character (nil) before #atEnd
> >> answers correctly.
> >>
> >> At the moment, ZnCharacterEncoder uses #atEnd on character / byte
> >> streams, so fails for stdin on a terminal.
> >>
> >> Back to your questions:
> >>
> >> On 11 April 2018 at 11:12, Sven Van Caekenberghe  wrote:
> >> >
> >> >
> >> >> On 11 Apr 2018, at 10:29, Alistair Grant 
> wrote:
> >> >>
> >> >> Hi Denis,
> >> >>
> >> >> On 11 April 2018 at 10:02, Denis Kudriashov 
> >> >> wrote:
> >> >>>
> >> >>> 2018-04-11 8:32 GMT+02:00 Alistair Grant :
> >> 
> >> >>> Where is it being said that #next and/or 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Denis Kudriashov]

Thanks for explanation.

I think it would be same scenario for socket stream where #atEnd is not
blocking. So I agree that it is expected behaviour.

The example is general enough to expect it to be working for any given pair
of in and out streams. So our streams should support this.


2018-04-11 14:56 GMT+02:00 Alistair Grant :

> Hi Sven & Dennis,
>
> On 11 April 2018 at 12:04, Denis Kudriashov  wrote:
> > Hi Alistair.
> >
> > I don't think anybody is annoyed by you. You are doing really good job.
> And
> > nice thing that you are super patient to continue :)
> >
>
> On 11 April 2018 at 12:13, Sven Van Caekenberghe  wrote:
> >
> > Yes, Alistair, you are a top notch open source contributor !
> >
> > For me, this discussion is about the difference between looking from low
> level details/issues/changes up, vs, from a higher level down.
>
> Thanks for your kind words.
>
>
>
> > What I try to understand is why blocking atEnd is bad?
> > Here is code from VMMaker:
> >
> > [stdin atEnd] whileFalse:
> > [| nextChunk |
> > stdout nextPutAll: 'squeak> '; flush.
> > nextChunk := stdin nextChunkNoTag.
> > [nextChunk notEmpty and: [nextChunk first isSeparator]] whileTrue:
> > [nextChunk := nextChunk allButFirst].
> > Transcript cr; nextPutAll: nextChunk; cr; flush.
> > [stdout print: (Compiler evaluate: nextChunk); cr; flush]
> > on: Error
> > do: [:ex| self logError: ex description inContext: ex signalerContext to:
> > stderr]].
> > quitOnEof ifTrue:
> > [SourceFiles at: 2 put: nil.
> > Smalltalk snapshot: false andQuit: true]
> >
> >
> > I am not see why it breaks with blocking #atEnd. Can you explain?
>
>
> First consider the case where #atEnd doesn't block and we just want to
> evaluate 4+3:
>
> 1. #atEnd will return false
> 2. the loop will print the prompt
> 3. wait for input (stdin nextChunkNoTag)
> 4. print the result
> 5. goto 1.
>
> So the screen will look like:
>
> squeak> 4+3!
> 7
> squeak> [cursor here]
>
> Which is what we expect (prompt, input, result, prompt).
>
> If #atEnd is blocking the VM will hang at step 1 until the user enters
> something in the terminal.  In Ubuntu at least terminal input appears to
> be line buffered, so for the example above the terminal will look like:
>
> 4+3!
> squeak> 7
> [cursor here]
>
> We don't get the prompt when the program is started, the result is
> printed after the prompt, and then there's just a cursor sitting at the
> start of the next line.
>
> Obviously the program could be re-written to have the correct output
> with #atEnd blocking.  But I'm arguing that this program is
> representative of many others, and we don't want to break backward
> compatibility in this case.
>
> Cheers,
> Alistair
>
>
>
>
>
>
>
>
> > 2018-04-11 11:41 GMT+02:00 Alistair Grant :
> >>
> >> Hi Sven,
> >>
> >> Oh dear.  I feel as though I'm not getting my concerns across at all
> >> well, and I'm pushing hard enough that all I'm going to do is make
> >> people annoyed.  So let me try to restate the issue one last time
> >> before answering your questions directly.
> >>
> >> Pharo & Squeak have unwritten rules about stream usage that I suspect
> >> have just emerged over time without being designed.
> >>
> >> If you want to be able to iterate over any stream, and in particular
> >> stdin from a terminal (which, as far as I know, is the outlier that
> >> causes all the problems) you have to follow these rules:
> >>
> >> 1.  If the stream is character / byte oriented you have to check for
> >> EOF using "stream next == nil".  #atEnd can be used, but you'll still
> >> have to do the nil check.
> >>
> >> 2.  All other streams have to check for EOF (end of stream) using
> >> #atEnd.  "stream next == nil" can be used, but you'll still need to
> >> test #atEnd to determine whether nil is a value returned by the
> >> stream.
> >>
> >> If you write code that you want to be able to consume characters,
> >> bytes or any other object, you'll have to test both "stream next ==
> >> nil" and #atEnd.
> >>
> >> The rules are the result of the original blue book design being that
> >> #atEnd should be used, and then character input from a terminal being
> >> added later, but always returning an EOF character (nil) before #atEnd
> >> answers correctly.
> >>
> >> At the moment, ZnCharacterEncoder uses #atEnd on character / byte
> >> streams, so fails for stdin on a terminal.
> >>
> >> Back to your questions:
> >>
> >> On 11 April 2018 at 11:12, Sven Van Caekenberghe  wrote:
> >> >
> >> >
> >> >> On 11 Apr 2018, at 10:29, Alistair Grant 
> wrote:
> >> >>
> >> >> Hi Denis,
> >> >>
> >> >> On 11 April 2018 at 10:02, Denis Kudriashov 
> >> >> wrote:
> >> >>>
> >> >>> 2018-04-11 8:32 GMT+02:00 Alistair Grant :
> >> 
> >> >>> Where is it being said that #next and/or #atEnd should be
> blocking
> >> >>> or
> >> >>> non-blocking ?
> 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

> On 11 Apr 2018, at 11:12, Sven Van Caekenberghe  wrote:
> 
> How does one modify #atEnd to block ? I suppose you are talking about 
> StdioStream>>#atEnd ?
> 
> ^ self peek isNil
> 
> ?

Still the same question, how do you implement a blocking #atEnd for stdin ?

I have seen your stdio.cs which is indeed needed as the current 
StdioStream>>#atEnd is bogus for sure.

But that is still a non-blocking one, right ?

Since there is a peekBuffer in StdioStream, why can't that be used ?

I have run your example testAtEnd.st now, and it works/fails as advertised.

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

Hi Sven & Dennis,

On 11 April 2018 at 12:04, Denis Kudriashov  wrote:
> Hi Alistair.
>
> I don't think anybody is annoyed by you. You are doing really good job. And
> nice thing that you are super patient to continue :)
>

On 11 April 2018 at 12:13, Sven Van Caekenberghe  wrote:
>
> Yes, Alistair, you are a top notch open source contributor !
>
> For me, this discussion is about the difference between looking from low 
> level details/issues/changes up, vs, from a higher level down.

Thanks for your kind words.



> What I try to understand is why blocking atEnd is bad?
> Here is code from VMMaker:
>
> [stdin atEnd] whileFalse:
> [| nextChunk |
> stdout nextPutAll: 'squeak> '; flush.
> nextChunk := stdin nextChunkNoTag.
> [nextChunk notEmpty and: [nextChunk first isSeparator]] whileTrue:
> [nextChunk := nextChunk allButFirst].
> Transcript cr; nextPutAll: nextChunk; cr; flush.
> [stdout print: (Compiler evaluate: nextChunk); cr; flush]
> on: Error
> do: [:ex| self logError: ex description inContext: ex signalerContext to:
> stderr]].
> quitOnEof ifTrue:
> [SourceFiles at: 2 put: nil.
> Smalltalk snapshot: false andQuit: true]
>
>
> I am not see why it breaks with blocking #atEnd. Can you explain?


First consider the case where #atEnd doesn't block and we just want to
evaluate 4+3:

1. #atEnd will return false
2. the loop will print the prompt
3. wait for input (stdin nextChunkNoTag)
4. print the result
5. goto 1.

So the screen will look like:

squeak> 4+3!
7
squeak> [cursor here]

Which is what we expect (prompt, input, result, prompt).

If #atEnd is blocking the VM will hang at step 1 until the user enters
something in the terminal.  In Ubuntu at least terminal input appears to
be line buffered, so for the example above the terminal will look like:

4+3!
squeak> 7
[cursor here]

We don't get the prompt when the program is started, the result is
printed after the prompt, and then there's just a cursor sitting at the
start of the next line.

Obviously the program could be re-written to have the correct output
with #atEnd blocking.  But I'm arguing that this program is
representative of many others, and we don't want to break backward
compatibility in this case.

Cheers,
Alistair








> 2018-04-11 11:41 GMT+02:00 Alistair Grant :
>>
>> Hi Sven,
>>
>> Oh dear.  I feel as though I'm not getting my concerns across at all
>> well, and I'm pushing hard enough that all I'm going to do is make
>> people annoyed.  So let me try to restate the issue one last time
>> before answering your questions directly.
>>
>> Pharo & Squeak have unwritten rules about stream usage that I suspect
>> have just emerged over time without being designed.
>>
>> If you want to be able to iterate over any stream, and in particular
>> stdin from a terminal (which, as far as I know, is the outlier that
>> causes all the problems) you have to follow these rules:
>>
>> 1.  If the stream is character / byte oriented you have to check for
>> EOF using "stream next == nil".  #atEnd can be used, but you'll still
>> have to do the nil check.
>>
>> 2.  All other streams have to check for EOF (end of stream) using
>> #atEnd.  "stream next == nil" can be used, but you'll still need to
>> test #atEnd to determine whether nil is a value returned by the
>> stream.
>>
>> If you write code that you want to be able to consume characters,
>> bytes or any other object, you'll have to test both "stream next ==
>> nil" and #atEnd.
>>
>> The rules are the result of the original blue book design being that
>> #atEnd should be used, and then character input from a terminal being
>> added later, but always returning an EOF character (nil) before #atEnd
>> answers correctly.
>>
>> At the moment, ZnCharacterEncoder uses #atEnd on character / byte
>> streams, so fails for stdin on a terminal.
>>
>> Back to your questions:
>>
>> On 11 April 2018 at 11:12, Sven Van Caekenberghe  wrote:
>> >
>> >
>> >> On 11 Apr 2018, at 10:29, Alistair Grant  wrote:
>> >>
>> >> Hi Denis,
>> >>
>> >> On 11 April 2018 at 10:02, Denis Kudriashov 
>> >> wrote:
>> >>>
>> >>> 2018-04-11 8:32 GMT+02:00 Alistair Grant :
>> 
>> >>> Where is it being said that #next and/or #atEnd should be blocking
>> >>> or
>> >>> non-blocking ?
>> >>
>> >> There is existing code that assumes that #atEnd is non-blocking and
>> >> that #next is allowed block.  I believe that we should keep those
>> >> conditions.
>> >
>> > I fail to see where that is written down, either way. Can you point
>> > me
>> > to comments stating that, I would really like to know ?
>> 
>>  I'm not aware of it being written down, just that ever existing
>>  implementation I'm aware of behaves this way.
>> 
>>  On the other hand, making #atEnd blocking breaks Eliot's REPL sample
>>  (in Squeak).
>> >>>
>> >>>
>> >>> Could 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

> On 11 Apr 2018, at 12:04, Denis Kudriashov  wrote:
> 
> I don't think anybody is annoyed by you. You are doing really good job. And 
> nice thing that you are super patient to continue :)

Yes, Alistair, you are a top notch open source contributor !

For me, this discussion is about the difference between looking from low level 
details/issues/changes up, vs, from a higher level down.

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Denis Kudriashov]

Hi Alistair.

I don't think anybody is annoyed by you. You are doing really good job. And
nice thing that you are super patient to continue :)

What I try to understand is why blocking atEnd is bad?
Here is code from VMMaker:

[stdin atEnd] whileFalse:
[| nextChunk |
stdout nextPutAll: 'squeak> '; flush.
nextChunk := stdin nextChunkNoTag.
[nextChunk notEmpty and: [nextChunk first isSeparator]] whileTrue:
[nextChunk := nextChunk allButFirst].
Transcript cr; nextPutAll: nextChunk; cr; flush.
[stdout print: (Compiler evaluate: nextChunk); cr; flush]
on: Error
do: [:ex| self logError: ex description inContext: ex signalerContext to:
stderr]].
quitOnEof ifTrue:
[SourceFiles at: 2 put: nil.
Smalltalk snapshot: false andQuit: true]


I am not see why it breaks with blocking #atEnd. Can you explain?

2018-04-11 11:41 GMT+02:00 Alistair Grant :

> Hi Sven,
>
> Oh dear.  I feel as though I'm not getting my concerns across at all
> well, and I'm pushing hard enough that all I'm going to do is make
> people annoyed.  So let me try to restate the issue one last time
> before answering your questions directly.
>
> Pharo & Squeak have unwritten rules about stream usage that I suspect
> have just emerged over time without being designed.
>
> If you want to be able to iterate over any stream, and in particular
> stdin from a terminal (which, as far as I know, is the outlier that
> causes all the problems) you have to follow these rules:
>
> 1.  If the stream is character / byte oriented you have to check for
> EOF using "stream next == nil".  #atEnd can be used, but you'll still
> have to do the nil check.
>
> 2.  All other streams have to check for EOF (end of stream) using
> #atEnd.  "stream next == nil" can be used, but you'll still need to
> test #atEnd to determine whether nil is a value returned by the
> stream.
>
> If you write code that you want to be able to consume characters,
> bytes or any other object, you'll have to test both "stream next ==
> nil" and #atEnd.
>
> The rules are the result of the original blue book design being that
> #atEnd should be used, and then character input from a terminal being
> added later, but always returning an EOF character (nil) before #atEnd
> answers correctly.
>
> At the moment, ZnCharacterEncoder uses #atEnd on character / byte
> streams, so fails for stdin on a terminal.
>
> Back to your questions:
>
> On 11 April 2018 at 11:12, Sven Van Caekenberghe  wrote:
> >
> >
> >> On 11 Apr 2018, at 10:29, Alistair Grant  wrote:
> >>
> >> Hi Denis,
> >>
> >> On 11 April 2018 at 10:02, Denis Kudriashov 
> wrote:
> >>>
> >>> 2018-04-11 8:32 GMT+02:00 Alistair Grant :
> 
> >>> Where is it being said that #next and/or #atEnd should be blocking
> or
> >>> non-blocking ?
> >>
> >> There is existing code that assumes that #atEnd is non-blocking and
> >> that #next is allowed block.  I believe that we should keep those
> >> conditions.
> >
> > I fail to see where that is written down, either way. Can you point
> me
> > to comments stating that, I would really like to know ?
> 
>  I'm not aware of it being written down, just that ever existing
>  implementation I'm aware of behaves this way.
> 
>  On the other hand, making #atEnd blocking breaks Eliot's REPL sample
>  (in Squeak).
> >>>
> >>>
> >>> Could you write here this example, please?
> >>
> >> The code is loaded in squeak using:
> >>
> >> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/
> buildspurtrunkreaderimage.sh
> >>
> >> for 32 bit images.  It loads:
> >>
> >> https://github.com/OpenSmalltalk/opensmalltalk-
> vm/blob/Cog/image/LoadReader.st
> >>
> >> which loads package CogTools-Listener in http://source.squeak.org/
> VMMaker
> >>
> >> An image that automatically runs the code and nothing else is created
> in:
> >>
> >> https://github.com/OpenSmalltalk/opensmalltalk-
> vm/blob/Cog/image/StartReader.st
> >>
> >>
> >> If you want to run it interactively you can load CogTools-Listener and
> >> do something like:
> >>
> >> StdioListener new
> >>quitOnEof: false;
> >>run
> >
> > What does #quitOnEof: do ? Can the StdioListener code be browsed/viewed
> online somewhere ?
>
> I just referenced this as an example of making #atEnd (really
> FilePlugin>>primitiveFileAtEnd) blocking causing problems.  I wasn't
> expecting people to go and look at the code or use it as a test.
>
> If you really want to look at it (from Pharo):
>
> 1. Add http://source.squeak.org/VMMaker as a repository.
> 2. Browse the CogTools-Listener package
>
>
> >> If you modify #atEnd to block it will result in the "squeak>" input
> >> prompt being printed in the terminal after the input has been entered.
> >
> > How does one modify #atEnd to block ? I suppose you are talking about
> StdioStream>>#atEnd ?
>
> I meant the primitive, i.e. FilePlugin>>primitiveFileAtEnd /
> 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

Hi Sven,

Oh dear.  I feel as though I'm not getting my concerns across at all
well, and I'm pushing hard enough that all I'm going to do is make
people annoyed.  So let me try to restate the issue one last time
before answering your questions directly.

Pharo & Squeak have unwritten rules about stream usage that I suspect
have just emerged over time without being designed.

If you want to be able to iterate over any stream, and in particular
stdin from a terminal (which, as far as I know, is the outlier that
causes all the problems) you have to follow these rules:

1.  If the stream is character / byte oriented you have to check for
EOF using "stream next == nil".  #atEnd can be used, but you'll still
have to do the nil check.

2.  All other streams have to check for EOF (end of stream) using
#atEnd.  "stream next == nil" can be used, but you'll still need to
test #atEnd to determine whether nil is a value returned by the
stream.

If you write code that you want to be able to consume characters,
bytes or any other object, you'll have to test both "stream next ==
nil" and #atEnd.

The rules are the result of the original blue book design being that
#atEnd should be used, and then character input from a terminal being
added later, but always returning an EOF character (nil) before #atEnd
answers correctly.

At the moment, ZnCharacterEncoder uses #atEnd on character / byte
streams, so fails for stdin on a terminal.

Back to your questions:

On 11 April 2018 at 11:12, Sven Van Caekenberghe  wrote:
>
>
>> On 11 Apr 2018, at 10:29, Alistair Grant  wrote:
>>
>> Hi Denis,
>>
>> On 11 April 2018 at 10:02, Denis Kudriashov  wrote:
>>>
>>> 2018-04-11 8:32 GMT+02:00 Alistair Grant :

>>> Where is it being said that #next and/or #atEnd should be blocking or
>>> non-blocking ?
>>
>> There is existing code that assumes that #atEnd is non-blocking and
>> that #next is allowed block.  I believe that we should keep those
>> conditions.
>
> I fail to see where that is written down, either way. Can you point me
> to comments stating that, I would really like to know ?

 I'm not aware of it being written down, just that ever existing
 implementation I'm aware of behaves this way.

 On the other hand, making #atEnd blocking breaks Eliot's REPL sample
 (in Squeak).
>>>
>>>
>>> Could you write here this example, please?
>>
>> The code is loaded in squeak using:
>>
>> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/buildspurtrunkreaderimage.sh
>>
>> for 32 bit images.  It loads:
>>
>> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/LoadReader.st
>>
>> which loads package CogTools-Listener in http://source.squeak.org/VMMaker
>>
>> An image that automatically runs the code and nothing else is created in:
>>
>> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/StartReader.st
>>
>>
>> If you want to run it interactively you can load CogTools-Listener and
>> do something like:
>>
>> StdioListener new
>>quitOnEof: false;
>>run
>
> What does #quitOnEof: do ? Can the StdioListener code be browsed/viewed 
> online somewhere ?

I just referenced this as an example of making #atEnd (really
FilePlugin>>primitiveFileAtEnd) blocking causing problems.  I wasn't
expecting people to go and look at the code or use it as a test.

If you really want to look at it (from Pharo):

1. Add http://source.squeak.org/VMMaker as a repository.
2. Browse the CogTools-Listener package


>> If you modify #atEnd to block it will result in the "squeak>" input
>> prompt being printed in the terminal after the input has been entered.
>
> How does one modify #atEnd to block ? I suppose you are talking about 
> StdioStream>>#atEnd ?

I meant the primitive, i.e. FilePlugin>>primitiveFileAtEnd /
FilePluginPrims>>atEnd:.


>  ^ self peek isNil
>
> ?
>
> PS: I liked your runnable example better, I will try it later on. Thx!

Right.  My code is meant to be minimal and trigger the problem I'm
actually focused on - that ZnCharacterEncoder doesn't work with stdin
from a terminal.

Sven has expressed a hesitation to change the internal operation of
the Zinc streams from using #atEnd to "stream peek == nil" and this
whole discussion is really about us trying to resolve our different
perspective of the best path forward.  I respect Sven and his work so
I'm trying to justify the change (but I'm not expressing it at all
well, obviously).

Cheers,
Alistair



>> The code can be loaded in to Pharo and basically works, but the output
>> tends to be hidden behind the next input prompt because it uses #cr
>> instead of #lf.  You can easily modify StdioListener>>initialize to
>> set the line end convention in stdout.
>>
>> NOTE: It is not intended to be a release quality implementation of a
>> evaluation loop.  The whole purpose as I understand it is for it to be
>> as simple as 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

> On 11 Apr 2018, at 10:29, Alistair Grant  wrote:
> 
> Hi Denis,
> 
> On 11 April 2018 at 10:02, Denis Kudriashov  wrote:
>> 
>> 2018-04-11 8:32 GMT+02:00 Alistair Grant :
>>> 
>> Where is it being said that #next and/or #atEnd should be blocking or
>> non-blocking ?
> 
> There is existing code that assumes that #atEnd is non-blocking and
> that #next is allowed block.  I believe that we should keep those
> conditions.
 
 I fail to see where that is written down, either way. Can you point me
 to comments stating that, I would really like to know ?
>>> 
>>> I'm not aware of it being written down, just that ever existing
>>> implementation I'm aware of behaves this way.
>>> 
>>> On the other hand, making #atEnd blocking breaks Eliot's REPL sample
>>> (in Squeak).
>> 
>> 
>> Could you write here this example, please?
> 
> The code is loaded in squeak using:
> 
> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/buildspurtrunkreaderimage.sh
> 
> for 32 bit images.  It loads:
> 
> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/LoadReader.st
> 
> which loads package CogTools-Listener in http://source.squeak.org/VMMaker
> 
> An image that automatically runs the code and nothing else is created in:
> 
> https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/StartReader.st
> 
> 
> If you want to run it interactively you can load CogTools-Listener and
> do something like:
> 
> StdioListener new
>quitOnEof: false;
>run

What does #quitOnEof: do ? Can the StdioListener code be browsed/viewed online 
somewhere ?

> If you modify #atEnd to block it will result in the "squeak>" input
> prompt being printed in the terminal after the input has been entered.

How does one modify #atEnd to block ? I suppose you are talking about 
StdioStream>>#atEnd ?

 ^ self peek isNil

?

PS: I liked your runnable example better, I will try it later on. Thx!

> The code can be loaded in to Pharo and basically works, but the output
> tends to be hidden behind the next input prompt because it uses #cr
> instead of #lf.  You can easily modify StdioListener>>initialize to
> set the line end convention in stdout.
> 
> NOTE: It is not intended to be a release quality implementation of a
> evaluation loop.  The whole purpose as I understand it is for it to be
> as simple as possible to assist in tracking down issues using the VM
> simulator.  It runs minimal code to get to the point of waiting for
> user input and then allows an expression that causes problems to be
> entered and traced using the simulator.
> 
> Cheers,
> Alistair
> 
> 
> 
>> How is this related to how EOF is signalled ?
> 
> Because, combined with terminal EOF not being known until the user
> explicitly flags it (with Ctrl-D) it means that #atEnd can't be used
> for iterating over input from stdin connected to a terminal.
 
 This seems to me like an exception that only holds for one particular
 stream in one particular scenario (interactive stdin). I might be wrong.
 
>> It seems to me that there are quite a few classes of streams that are
>> 'special' in the sense that #next could be blocking and/or #atEnd could 
>> be
>> unclear - socket/network streams, serial streams, maybe stdio 
>> (interactive
>> or not). Without a message like #isDataAvailable you cannot handle those
>> without blocking.
> 
> Right.  I think this is a distraction (I was trying to explain some
> details, but it's causing more confusion instead of helping).
> 
> The important point is that #atEnd doesn't work for iterating over
> streams with terminal input
 
 Maybe you should also point to the actual code that fails. I mean you
 showed a partial stack trace, but not how you got there, precisely. How 
 does
 the application reading from an interactive stdin do to get into trouble ?
>>> 
>>> Included below.
>>> 
>>> 
>> Reading from stdin seems like a very rare case for a Smalltalk system
>> (not that it should not be possible).
> 
> There's been quite a bit of discussion and several projects recently
> related to using pharo for scripting, so it may become more common.
> E.g.
> 
> 
> https://www.quora.com/Can-Smalltalk-be-a-batch-file-scripting-language/answer/Philippe-Back-1?share=c19bfc95
> https://github.com/rajula96reddy/pharo-cli
 
 Still, it is not common at all.
 
>> I have a feeling that too much functionality is being pushed into too
>> small an API.
> 
> This is just about how should Zinc streams be iterating over the
> underlying streams.  You didn't like checking the result of #next for
> nil since it isn't general, correctly pointing out that nil is a valid
> value for non-byte oriented streams.  But #atEnd doesn't work for
> stdin from a terminal.

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

Hi Denis,

On 11 April 2018 at 10:02, Denis Kudriashov  wrote:
>
> 2018-04-11 8:32 GMT+02:00 Alistair Grant :
>>
>> >>> Where is it being said that #next and/or #atEnd should be blocking or
>> >>> non-blocking ?
>> >>
>> >> There is existing code that assumes that #atEnd is non-blocking and
>> >> that #next is allowed block.  I believe that we should keep those
>> >> conditions.
>> >
>> > I fail to see where that is written down, either way. Can you point me
>> > to comments stating that, I would really like to know ?
>>
>> I'm not aware of it being written down, just that ever existing
>> implementation I'm aware of behaves this way.
>>
>> On the other hand, making #atEnd blocking breaks Eliot's REPL sample
>> (in Squeak).
>
>
> Could you write here this example, please?

The code is loaded in squeak using:

https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/buildspurtrunkreaderimage.sh

for 32 bit images.  It loads:

https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/LoadReader.st

which loads package CogTools-Listener in http://source.squeak.org/VMMaker

An image that automatically runs the code and nothing else is created in:

https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/image/StartReader.st


If you want to run it interactively you can load CogTools-Listener and
do something like:

StdioListener new
quitOnEof: false;
run

If you modify #atEnd to block it will result in the "squeak>" input
prompt being printed in the terminal after the input has been entered.

The code can be loaded in to Pharo and basically works, but the output
tends to be hidden behind the next input prompt because it uses #cr
instead of #lf.  You can easily modify StdioListener>>initialize to
set the line end convention in stdout.

NOTE: It is not intended to be a release quality implementation of a
evaluation loop.  The whole purpose as I understand it is for it to be
as simple as possible to assist in tracking down issues using the VM
simulator.  It runs minimal code to get to the point of waiting for
user input and then allows an expression that causes problems to be
entered and traced using the simulator.

Cheers,
Alistair



>> >>> How is this related to how EOF is signalled ?
>> >>
>> >> Because, combined with terminal EOF not being known until the user
>> >> explicitly flags it (with Ctrl-D) it means that #atEnd can't be used
>> >> for iterating over input from stdin connected to a terminal.
>> >
>> > This seems to me like an exception that only holds for one particular
>> > stream in one particular scenario (interactive stdin). I might be wrong.
>> >
>> >>> It seems to me that there are quite a few classes of streams that are
>> >>> 'special' in the sense that #next could be blocking and/or #atEnd could 
>> >>> be
>> >>> unclear - socket/network streams, serial streams, maybe stdio 
>> >>> (interactive
>> >>> or not). Without a message like #isDataAvailable you cannot handle those
>> >>> without blocking.
>> >>
>> >> Right.  I think this is a distraction (I was trying to explain some
>> >> details, but it's causing more confusion instead of helping).
>> >>
>> >> The important point is that #atEnd doesn't work for iterating over
>> >> streams with terminal input
>> >
>> > Maybe you should also point to the actual code that fails. I mean you
>> > showed a partial stack trace, but not how you got there, precisely. How 
>> > does
>> > the application reading from an interactive stdin do to get into trouble ?
>>
>> Included below.
>>
>>
>> >>> Reading from stdin seems like a very rare case for a Smalltalk system
>> >>> (not that it should not be possible).
>> >>
>> >> There's been quite a bit of discussion and several projects recently
>> >> related to using pharo for scripting, so it may become more common.
>> >> E.g.
>> >>
>> >>
>> >> https://www.quora.com/Can-Smalltalk-be-a-batch-file-scripting-language/answer/Philippe-Back-1?share=c19bfc95
>> >> https://github.com/rajula96reddy/pharo-cli
>> >
>> > Still, it is not common at all.
>> >
>> >>> I have a feeling that too much functionality is being pushed into too
>> >>> small an API.
>> >>
>> >> This is just about how should Zinc streams be iterating over the
>> >> underlying streams.  You didn't like checking the result of #next for
>> >> nil since it isn't general, correctly pointing out that nil is a valid
>> >> value for non-byte oriented streams.  But #atEnd doesn't work for
>> >> stdin from a terminal.
>> >>
>> >>
>> >> At this point I think there are three options:
>> >>
>> >> 1. Modify Zinc to check the return value of #next instead of using
>> >> #atEnd.
>> >>
>> >> This is what all existing character / byte oriented streams in Squeak
>> >> and Pharo do.  At that point the Zinc streams can be used on all file
>> >> / stdio input and output.
>> >
>> > I agree that such code exists in many places, but there is lots of
>> > stream reading that does not check for nils.
>>
>> 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Denis Kudriashov]

2018-04-11 8:32 GMT+02:00 Alistair Grant :

> >>> Where is it being said that #next and/or #atEnd should be blocking or
> non-blocking ?
> >>
> >> There is existing code that assumes that #atEnd is non-blocking and
> >> that #next is allowed block.  I believe that we should keep those
> >> conditions.
> >
> > I fail to see where that is written down, either way. Can you point me
> to comments stating that, I would really like to know ?
>
> I'm not aware of it being written down, just that ever existing
> implementation I'm aware of behaves this way.
>
> On the other hand, making #atEnd blocking breaks Eliot's REPL sample
> (in Squeak).
>

Could you write here this example, please?


>
>
>
> >>> How is this related to how EOF is signalled ?
> >>
> >> Because, combined with terminal EOF not being known until the user
> >> explicitly flags it (with Ctrl-D) it means that #atEnd can't be used
> >> for iterating over input from stdin connected to a terminal.
> >
> > This seems to me like an exception that only holds for one particular
> stream in one particular scenario (interactive stdin). I might be wrong.
> >
> >>> It seems to me that there are quite a few classes of streams that are
> 'special' in the sense that #next could be blocking and/or #atEnd could be
> unclear - socket/network streams, serial streams, maybe stdio (interactive
> or not). Without a message like #isDataAvailable you cannot handle those
> without blocking.
> >>
> >> Right.  I think this is a distraction (I was trying to explain some
> >> details, but it's causing more confusion instead of helping).
> >>
> >> The important point is that #atEnd doesn't work for iterating over
> >> streams with terminal input
> >
> > Maybe you should also point to the actual code that fails. I mean you
> showed a partial stack trace, but not how you got there, precisely. How
> does the application reading from an interactive stdin do to get into
> trouble ?
>
> Included below.
>
>
> >>> Reading from stdin seems like a very rare case for a Smalltalk system
> (not that it should not be possible).
> >>
> >> There's been quite a bit of discussion and several projects recently
> >> related to using pharo for scripting, so it may become more common.
> >> E.g.
> >>
> >> https://www.quora.com/Can-Smalltalk-be-a-batch-file-
> scripting-language/answer/Philippe-Back-1?share=c19bfc95
> >> https://github.com/rajula96reddy/pharo-cli
> >
> > Still, it is not common at all.
> >
> >>> I have a feeling that too much functionality is being pushed into too
> small an API.
> >>
> >> This is just about how should Zinc streams be iterating over the
> >> underlying streams.  You didn't like checking the result of #next for
> >> nil since it isn't general, correctly pointing out that nil is a valid
> >> value for non-byte oriented streams.  But #atEnd doesn't work for
> >> stdin from a terminal.
> >>
> >>
> >> At this point I think there are three options:
> >>
> >> 1. Modify Zinc to check the return value of #next instead of using
> #atEnd.
> >>
> >> This is what all existing character / byte oriented streams in Squeak
> >> and Pharo do.  At that point the Zinc streams can be used on all file
> >> / stdio input and output.
> >
> > I agree that such code exists in many places, but there is lots of
> stream reading that does not check for nils.
>
> Right.  Streams can be categorised in many ways, but for this
> discussion I think streams are broken in to two types:
>
> 1) Byte / Character oriented
> 2) All others
>
> For historical reasons, byte / character oriented streams need to
> check for EOF by using "stream next == nil" and all other streams
> should use #atEnd.
>
> This avoids the "nil being part of the domain" issue that was
> discussed earlier in the thread.
>
>
> >> 2. Modify all streams to signal EOF in some other way, i.e. a sentinel
> >> or notification / exception.
> >>
> >> This is what we were discussing below.  But it is a decent chunk of
> >> work with significant impact on the existing code base.
> >
> > Agreed. This would be a future extension.
> >
> >> 3. Require anyone who wants to read from stdin to code around Zinc's
> >> inability to handle terminal input.
> >>
> >> I'd prefer to avoid this option if possible.
> >
> > See higher for a more concrete usage example request.
>
>
> testAtEnd.st
> --
> | ch stream string stdin |
>
> 'stdio.cs' asFileReference fileIn.
> "stdin := FileStream stdin."
> stdin := ZnCharacterReadStream on:
> (ZnBufferedReadStream on:
> Stdio stdin).
> stream := (String new: 100) writeStream.
> ch := stdin next.
> [ ch == nil ] whileFalse: [
> stream nextPut: ch.
> ch := stdin next. ].
> string := stream contents.
> FileStream stdout
> nextPutAll: string; lf;
> nextPutAll: 'Characters read: ';
> nextPutAll: string size asString;
> lf.
> Smalltalk snapshot: false andQuit: true.
> --
>
> Execute with:
>
> ./pharo --headless Pharo7.0-64bit-e76f1a2.image testAtEnd.st
>
> and type 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

Hi Sven,

On 10 April 2018 at 23:54, Sven Van Caekenberghe  wrote:
> Alistair,
>
> I am replying in-between, please don't take my remarks as being negative, I 
> just like to straighten things out as clear as possible.

No problem.


>> On 10 Apr 2018, at 22:14, Alistair Grant  wrote:
>>
>> Hi Sven,
>>
>> On 10 April 2018 at 19:36, Sven Van Caekenberghe  wrote:
>>> I have trouble understanding your problem analysis, and how your proposed 
>>> solution, would solve it.
>>
>> The discussion started with my proposal to modify the Zinc streams to
>> check the return value of #next for nil rather than using #atEnd to
>> iterate over the underlying stream.  You correctly pointed out that
>> the intention behind #atEnd is that it can be used to control
>> iterating over streams.
>
> OK
>
>>> Where is it being said that #next and/or #atEnd should be blocking or 
>>> non-blocking ?
>>
>> There is existing code that assumes that #atEnd is non-blocking and
>> that #next is allowed block.  I believe that we should keep those
>> conditions.
>
> I fail to see where that is written down, either way. Can you point me to 
> comments stating that, I would really like to know ?

I'm not aware of it being written down, just that ever existing
implementation I'm aware of behaves this way.

On the other hand, making #atEnd blocking breaks Eliot's REPL sample
(in Squeak).



>>> How is this related to how EOF is signalled ?
>>
>> Because, combined with terminal EOF not being known until the user
>> explicitly flags it (with Ctrl-D) it means that #atEnd can't be used
>> for iterating over input from stdin connected to a terminal.
>
> This seems to me like an exception that only holds for one particular stream 
> in one particular scenario (interactive stdin). I might be wrong.
>
>>> It seems to me that there are quite a few classes of streams that are 
>>> 'special' in the sense that #next could be blocking and/or #atEnd could be 
>>> unclear - socket/network streams, serial streams, maybe stdio (interactive 
>>> or not). Without a message like #isDataAvailable you cannot handle those 
>>> without blocking.
>>
>> Right.  I think this is a distraction (I was trying to explain some
>> details, but it's causing more confusion instead of helping).
>>
>> The important point is that #atEnd doesn't work for iterating over
>> streams with terminal input
>
> Maybe you should also point to the actual code that fails. I mean you showed 
> a partial stack trace, but not how you got there, precisely. How does the 
> application reading from an interactive stdin do to get into trouble ?

Included below.


>>> Reading from stdin seems like a very rare case for a Smalltalk system (not 
>>> that it should not be possible).
>>
>> There's been quite a bit of discussion and several projects recently
>> related to using pharo for scripting, so it may become more common.
>> E.g.
>>
>> https://www.quora.com/Can-Smalltalk-be-a-batch-file-scripting-language/answer/Philippe-Back-1?share=c19bfc95
>> https://github.com/rajula96reddy/pharo-cli
>
> Still, it is not common at all.
>
>>> I have a feeling that too much functionality is being pushed into too small 
>>> an API.
>>
>> This is just about how should Zinc streams be iterating over the
>> underlying streams.  You didn't like checking the result of #next for
>> nil since it isn't general, correctly pointing out that nil is a valid
>> value for non-byte oriented streams.  But #atEnd doesn't work for
>> stdin from a terminal.
>>
>>
>> At this point I think there are three options:
>>
>> 1. Modify Zinc to check the return value of #next instead of using #atEnd.
>>
>> This is what all existing character / byte oriented streams in Squeak
>> and Pharo do.  At that point the Zinc streams can be used on all file
>> / stdio input and output.
>
> I agree that such code exists in many places, but there is lots of stream 
> reading that does not check for nils.

Right.  Streams can be categorised in many ways, but for this
discussion I think streams are broken in to two types:

1) Byte / Character oriented
2) All others

For historical reasons, byte / character oriented streams need to
check for EOF by using "stream next == nil" and all other streams
should use #atEnd.

This avoids the "nil being part of the domain" issue that was
discussed earlier in the thread.


>> 2. Modify all streams to signal EOF in some other way, i.e. a sentinel
>> or notification / exception.
>>
>> This is what we were discussing below.  But it is a decent chunk of
>> work with significant impact on the existing code base.
>
> Agreed. This would be a future extension.
>
>> 3. Require anyone who wants to read from stdin to code around Zinc's
>> inability to handle terminal input.
>>
>> I'd prefer to avoid this option if possible.
>
> See higher for a more concrete usage example request.


testAtEnd.st
--
| ch stream string stdin |

'stdio.cs' asFileReference fileIn.
"stdin := FileStream 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

Alistair,

I am replying in-between, please don't take my remarks as being negative, I 
just like to straighten things out as clear as possible.

> On 10 Apr 2018, at 22:14, Alistair Grant  wrote:
> 
> Hi Sven,
> 
> On 10 April 2018 at 19:36, Sven Van Caekenberghe  wrote:
>> I have trouble understanding your problem analysis, and how your proposed 
>> solution, would solve it.
> 
> The discussion started with my proposal to modify the Zinc streams to
> check the return value of #next for nil rather than using #atEnd to
> iterate over the underlying stream.  You correctly pointed out that
> the intention behind #atEnd is that it can be used to control
> iterating over streams.

OK

>> Where is it being said that #next and/or #atEnd should be blocking or 
>> non-blocking ?
> 
> There is existing code that assumes that #atEnd is non-blocking and
> that #next is allowed block.  I believe that we should keep those
> conditions.

I fail to see where that is written down, either way. Can you point me to 
comments stating that, I would really like to know ?

>> How is this related to how EOF is signalled ?
> 
> Because, combined with terminal EOF not being known until the user
> explicitly flags it (with Ctrl-D) it means that #atEnd can't be used
> for iterating over input from stdin connected to a terminal.

This seems to me like an exception that only holds for one particular stream in 
one particular scenario (interactive stdin). I might be wrong.

>> It seems to me that there are quite a few classes of streams that are 
>> 'special' in the sense that #next could be blocking and/or #atEnd could be 
>> unclear - socket/network streams, serial streams, maybe stdio (interactive 
>> or not). Without a message like #isDataAvailable you cannot handle those 
>> without blocking.
> 
> Right.  I think this is a distraction (I was trying to explain some
> details, but it's causing more confusion instead of helping).
> 
> The important point is that #atEnd doesn't work for iterating over
> streams with terminal input

Maybe you should also point to the actual code that fails. I mean you showed a 
partial stack trace, but not how you got there, precisely. How does the 
application reading from an interactive stdin do to get into trouble ?

>> Reading from stdin seems like a very rare case for a Smalltalk system (not 
>> that it should not be possible).
> 
> There's been quite a bit of discussion and several projects recently
> related to using pharo for scripting, so it may become more common.
> E.g.
> 
> https://www.quora.com/Can-Smalltalk-be-a-batch-file-scripting-language/answer/Philippe-Back-1?share=c19bfc95
> https://github.com/rajula96reddy/pharo-cli

Still, it is not common at all.

>> I have a feeling that too much functionality is being pushed into too small 
>> an API.
> 
> This is just about how should Zinc streams be iterating over the
> underlying streams.  You didn't like checking the result of #next for
> nil since it isn't general, correctly pointing out that nil is a valid
> value for non-byte oriented streams.  But #atEnd doesn't work for
> stdin from a terminal.
> 
> 
> At this point I think there are three options:
> 
> 1. Modify Zinc to check the return value of #next instead of using #atEnd.
> 
> This is what all existing character / byte oriented streams in Squeak
> and Pharo do.  At that point the Zinc streams can be used on all file
> / stdio input and output.

I agree that such code exists in many places, but there is lots of stream 
reading that does not check for nils. 

> 2. Modify all streams to signal EOF in some other way, i.e. a sentinel
> or notification / exception.
> 
> This is what we were discussing below.  But it is a decent chunk of
> work with significant impact on the existing code base.

Agreed. This would be a future extension.

> 3. Require anyone who wants to read from stdin to code around Zinc's
> inability to handle terminal input.
> 
> I'd prefer to avoid this option if possible.

See higher for a more concrete usage example request. 

> Does that clarify the situation?

Yes, it helps. Thanks. But questions remain.

> Thanks,
> Alistair
> 
> 
> 
>>> On 10 Apr 2018, at 18:30, Alistair Grant  wrote:
>>> 
>>> First a quick update:
>>> 
>>> After doing some work on primitiveFileAtEnd, #atEnd now answers
>>> correctly for files that don't report their size correctly, e.g.
>>> /dev/urandom and /proc/cpuinfo, whether the files are opened directly or
>>> redirected through stdin.
>>> 
>>> However determining whether stdin from a terminal has reached the end of
>>> file can't be done without making #atEnd blocking since we have to wait
>>> for the user to flag the end of file, e.g. by typing Ctrl-D.  And #atEnd
>>> is assumed to be non-blocking.
>>> 
>>> So currently using ZnCharacterReadStream with stdin from a terminal will
>>> result in a stack dump similar to:
>>> 
>>> MessageNotUnderstood: receiver of "<" is nil
>>> 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

Hi Sven,

On 10 April 2018 at 19:36, Sven Van Caekenberghe  wrote:
> I have trouble understanding your problem analysis, and how your proposed 
> solution, would solve it.

The discussion started with my proposal to modify the Zinc streams to
check the return value of #next for nil rather than using #atEnd to
iterate over the underlying stream.  You correctly pointed out that
the intention behind #atEnd is that it can be used to control
iterating over streams.


> Where is it being said that #next and/or #atEnd should be blocking or 
> non-blocking ?

There is existing code that assumes that #atEnd is non-blocking and
that #next is allowed block.  I believe that we should keep those
conditions.


> How is this related to how EOF is signalled ?

Because, combined with terminal EOF not being known until the user
explicitly flags it (with Ctrl-D) it means that #atEnd can't be used
for iterating over input from stdin connected to a terminal.



> It seems to me that there are quite a few classes of streams that are 
> 'special' in the sense that #next could be blocking and/or #atEnd could be 
> unclear - socket/network streams, serial streams, maybe stdio (interactive or 
> not). Without a message like #isDataAvailable you cannot handle those without 
> blocking.

Right.  I think this is a distraction (I was trying to explain some
details, but it's causing more confusion instead of helping).

The important point is that #atEnd doesn't work for iterating over
streams with terminal input



> Reading from stdin seems like a very rare case for a Smalltalk system (not 
> that it should not be possible).

There's been quite a bit of discussion and several projects recently
related to using pharo for scripting, so it may become more common.
E.g.

https://www.quora.com/Can-Smalltalk-be-a-batch-file-scripting-language/answer/Philippe-Back-1?share=c19bfc95
https://github.com/rajula96reddy/pharo-cli



> I have a feeling that too much functionality is being pushed into too small 
> an API.


This is just about how should Zinc streams be iterating over the
underlying streams.  You didn't like checking the result of #next for
nil since it isn't general, correctly pointing out that nil is a valid
value for non-byte oriented streams.  But #atEnd doesn't work for
stdin from a terminal.


At this point I think there are three options:

1. Modify Zinc to check the return value of #next instead of using #atEnd.

This is what all existing character / byte oriented streams in Squeak
and Pharo do.  At that point the Zinc streams can be used on all file
/ stdio input and output.


2. Modify all streams to signal EOF in some other way, i.e. a sentinel
or notification / exception.

This is what we were discussing below.  But it is a decent chunk of
work with significant impact on the existing code base.


3. Require anyone who wants to read from stdin to code around Zinc's
inability to handle terminal input.

I'd prefer to avoid this option if possible.


Does that clarify the situation?

Thanks,
Alistair



>> On 10 Apr 2018, at 18:30, Alistair Grant  wrote:
>>
>> First a quick update:
>>
>> After doing some work on primitiveFileAtEnd, #atEnd now answers
>> correctly for files that don't report their size correctly, e.g.
>> /dev/urandom and /proc/cpuinfo, whether the files are opened directly or
>> redirected through stdin.
>>
>> However determining whether stdin from a terminal has reached the end of
>> file can't be done without making #atEnd blocking since we have to wait
>> for the user to flag the end of file, e.g. by typing Ctrl-D.  And #atEnd
>> is assumed to be non-blocking.
>>
>> So currently using ZnCharacterReadStream with stdin from a terminal will
>> result in a stack dump similar to:
>>
>> MessageNotUnderstood: receiver of "<" is nil
>> UndefinedObject(Object)>>doesNotUnderstand: #<
>> ZnUTF8Encoder>>nextCodePointFromStream:
>> ZnUTF8Encoder(ZnCharacterEncoder)>>nextFromStream:
>> ZnCharacterReadStream>>nextElement
>> ZnCharacterReadStream(ZnEncodedReadStream)>>next
>> UndefinedObject>>DoIt
>>
>>
>> Going back through the various suggestions that have been made regarding
>> using a sentinel object vs. raising a notification / exception, my
>> (still to be polished) suggestion is to:
>>
>> 1. Add an endOfStream instance variable
>> 2. When the end of the stream is reached answer the value of the
>>   instance variable (i.e. the result of sending #value to the variable).
>> 3. The initial default value would be a block that raises a Deprecation
>>   warning and then returns nil.  This would allow existing code to
>>   function for a changeover period.
>> 4. At the end of the deprecation period the default value would be
>>   changed to a unique sentinel object which would answer itself as its
>>   #value.
>>
>> At any time users of the stream can set their own sentinel, including a
>> block that raises an exception.
>>
>>
>> Cheers,
>> Alistair
>>
>>
>> On 4 April 2018 at 19:24, 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

I have trouble understanding your problem analysis, and how your proposed 
solution, would solve it.

Where is it being said that #next and/or #atEnd should be blocking or 
non-blocking ?
How is this related to how EOF is signalled ?

It seems to me that there are quite a few classes of streams that are 'special' 
in the sense that #next could be blocking and/or #atEnd could be unclear - 
socket/network streams, serial streams, maybe stdio (interactive or not). 
Without a message like #isDataAvailable you cannot handle those without 
blocking.

Reading from stdin seems like a very rare case for a Smalltalk system (not that 
it should not be possible).

I have a feeling that too much functionality is being pushed into too small an 
API.

> On 10 Apr 2018, at 18:30, Alistair Grant  wrote:
> 
> First a quick update:
> 
> After doing some work on primitiveFileAtEnd, #atEnd now answers
> correctly for files that don't report their size correctly, e.g.
> /dev/urandom and /proc/cpuinfo, whether the files are opened directly or
> redirected through stdin.
> 
> However determining whether stdin from a terminal has reached the end of
> file can't be done without making #atEnd blocking since we have to wait
> for the user to flag the end of file, e.g. by typing Ctrl-D.  And #atEnd
> is assumed to be non-blocking.
> 
> So currently using ZnCharacterReadStream with stdin from a terminal will
> result in a stack dump similar to:
> 
> MessageNotUnderstood: receiver of "<" is nil
> UndefinedObject(Object)>>doesNotUnderstand: #<
> ZnUTF8Encoder>>nextCodePointFromStream:
> ZnUTF8Encoder(ZnCharacterEncoder)>>nextFromStream:
> ZnCharacterReadStream>>nextElement
> ZnCharacterReadStream(ZnEncodedReadStream)>>next
> UndefinedObject>>DoIt
> 
> 
> Going back through the various suggestions that have been made regarding
> using a sentinel object vs. raising a notification / exception, my
> (still to be polished) suggestion is to:
> 
> 1. Add an endOfStream instance variable
> 2. When the end of the stream is reached answer the value of the
>   instance variable (i.e. the result of sending #value to the variable).
> 3. The initial default value would be a block that raises a Deprecation
>   warning and then returns nil.  This would allow existing code to
>   function for a changeover period.
> 4. At the end of the deprecation period the default value would be
>   changed to a unique sentinel object which would answer itself as its
>   #value.
> 
> At any time users of the stream can set their own sentinel, including a
> block that raises an exception.
> 
> 
> Cheers,
> Alistair
> 
> 
> On 4 April 2018 at 19:24, Stephane Ducasse  wrote:
>> Thanks for this discussion.
>> 
>> On Wed, Apr 4, 2018 at 1:37 PM, Sven Van Caekenberghe  wrote:
>>> Alistair,
>>> 
>>> First off, thanks for the discussions and your contributions, I really 
>>> appreciate them.
>>> 
>>> But I want to have a discussion at the high level of the definition and 
>>> semantics of the stream API in Pharo.
>>> 
 On 4 Apr 2018, at 13:20, Alistair Grant  wrote:
 
 On 4 April 2018 at 12:56, Sven Van Caekenberghe  wrote:
> Playing a bit devil's advocate, the idea is that, in general,
> 
> [ stream atEnd] whileFalse: [ stream next. "..." ].
> 
> is no longer allowed ?
 
 It hasn't been allowed "forever" [1].  It's just been misused for
 almost as long.
 
 [1] Time began when stdio stream support was introduced. :-)
>>> 
>>> I am still not convinced. Another way to put it would be that the old 
>>> #atEnd or #upToEnd do not make sense for these streams and some new loop is 
>>> needed, based on a new test (it exists for socket streams already).
>>> 
>>> [ stream isDataAvailable ] whileTrue: [ stream next ]
>>> 
> And you want to replace it with
> 
> [ stream next ifNil: [ false ] ifNotNil: [ :x | "..." true ] whileTrue.
> 
> That is a pretty big change, no ?
 
 That's the way quite a bit of code already operates.
 
 As Denis pointed out, it's obviously problematic in the general sense,
 since nil can be embedded in non-byte oriented streams.  I suspect
 that in practice not many people write code that reads streams from
 both byte oriented and non-byte oriented streams.
>>> 
>>> Maybe yes, maybe no. As Denis' example shows there is a clear definition 
>>> problem.
>>> 
>>> And I do use streams of byte arrays or strings all the time, this is really 
>>> important. I want my parsers to work on all kinds of streams.
>>> 
> I think/feel like a proper EOF exception would be better, more correct.
> 
> [ [ stream next. "..." true ] on: EOF do: [ false ] ] whileTrue.
 
 I agree, but the email thread Nicolas pointed to raises some
 performance questions about this approach.  It should be
 straightforward to do a basic performance comparison which I'll get
 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

First a quick update:

After doing some work on primitiveFileAtEnd, #atEnd now answers
correctly for files that don't report their size correctly, e.g.
/dev/urandom and /proc/cpuinfo, whether the files are opened directly or
redirected through stdin.

However determining whether stdin from a terminal has reached the end of
file can't be done without making #atEnd blocking since we have to wait
for the user to flag the end of file, e.g. by typing Ctrl-D.  And #atEnd
is assumed to be non-blocking.

So currently using ZnCharacterReadStream with stdin from a terminal will
result in a stack dump similar to:

MessageNotUnderstood: receiver of "<" is nil
UndefinedObject(Object)>>doesNotUnderstand: #<
ZnUTF8Encoder>>nextCodePointFromStream:
ZnUTF8Encoder(ZnCharacterEncoder)>>nextFromStream:
ZnCharacterReadStream>>nextElement
ZnCharacterReadStream(ZnEncodedReadStream)>>next
UndefinedObject>>DoIt


Going back through the various suggestions that have been made regarding
using a sentinel object vs. raising a notification / exception, my
(still to be polished) suggestion is to:

1. Add an endOfStream instance variable
2. When the end of the stream is reached answer the value of the
   instance variable (i.e. the result of sending #value to the variable).
3. The initial default value would be a block that raises a Deprecation
   warning and then returns nil.  This would allow existing code to
   function for a changeover period.
4. At the end of the deprecation period the default value would be
   changed to a unique sentinel object which would answer itself as its
   #value.

At any time users of the stream can set their own sentinel, including a
block that raises an exception.


Cheers,
Alistair


On 4 April 2018 at 19:24, Stephane Ducasse  wrote:
> Thanks for this discussion.
>
> On Wed, Apr 4, 2018 at 1:37 PM, Sven Van Caekenberghe  wrote:
>> Alistair,
>>
>> First off, thanks for the discussions and your contributions, I really 
>> appreciate them.
>>
>> But I want to have a discussion at the high level of the definition and 
>> semantics of the stream API in Pharo.
>>
>>> On 4 Apr 2018, at 13:20, Alistair Grant  wrote:
>>>
>>> On 4 April 2018 at 12:56, Sven Van Caekenberghe  wrote:
 Playing a bit devil's advocate, the idea is that, in general,

 [ stream atEnd] whileFalse: [ stream next. "..." ].

 is no longer allowed ?
>>>
>>> It hasn't been allowed "forever" [1].  It's just been misused for
>>> almost as long.
>>>
>>> [1] Time began when stdio stream support was introduced. :-)
>>
>> I am still not convinced. Another way to put it would be that the old #atEnd 
>> or #upToEnd do not make sense for these streams and some new loop is needed, 
>> based on a new test (it exists for socket streams already).
>>
>> [ stream isDataAvailable ] whileTrue: [ stream next ]
>>
 And you want to replace it with

 [ stream next ifNil: [ false ] ifNotNil: [ :x | "..." true ] whileTrue.

 That is a pretty big change, no ?
>>>
>>> That's the way quite a bit of code already operates.
>>>
>>> As Denis pointed out, it's obviously problematic in the general sense,
>>> since nil can be embedded in non-byte oriented streams.  I suspect
>>> that in practice not many people write code that reads streams from
>>> both byte oriented and non-byte oriented streams.
>>
>> Maybe yes, maybe no. As Denis' example shows there is a clear definition 
>> problem.
>>
>> And I do use streams of byte arrays or strings all the time, this is really 
>> important. I want my parsers to work on all kinds of streams.
>>
 I think/feel like a proper EOF exception would be better, more correct.

 [ [ stream next. "..." true ] on: EOF do: [ false ] ] whileTrue.
>>>
>>> I agree, but the email thread Nicolas pointed to raises some
>>> performance questions about this approach.  It should be
>>> straightforward to do a basic performance comparison which I'll get
>>> around to if other objections aren't raised.
>>
>> Reading in bigger blocks, using #readInto:startingAt:count: (which is 
>> basically Unix's (2) Read sys call), would solve performance problems, I 
>> think.
>>
 Will we throw away #atEnd then ? Do we need it if we cannot use it ?
>>>
>>> Unix file i/o returns EOF if the end of file has been reach OR if an
>>> error occurs.  You should still check #atEnd after reading past the
>>> end of the file to make sure no error occurred.  Another part of the
>>> primitive change I'm proposing is to return additional information
>>> about what went wrong in the event of an error.
>>
>> I am sorry, but this kind of semantics (the OR) is way too complex at the 
>> general image level, it is too specific and based on certain underlying 
>> implementation details.
>>
>> Sven
>>
>>> We could modify the read primitive so that it fails if an error has
>>> occurred, and then #atEnd wouldn't be required.
>>>
>>> Cheers,
>>> Alistair
>>>
>>>

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Stephane Ducasse]

Thanks for this discussion.

On Wed, Apr 4, 2018 at 1:37 PM, Sven Van Caekenberghe  wrote:
> Alistair,
>
> First off, thanks for the discussions and your contributions, I really 
> appreciate them.
>
> But I want to have a discussion at the high level of the definition and 
> semantics of the stream API in Pharo.
>
>> On 4 Apr 2018, at 13:20, Alistair Grant  wrote:
>>
>> On 4 April 2018 at 12:56, Sven Van Caekenberghe  wrote:
>>> Playing a bit devil's advocate, the idea is that, in general,
>>>
>>> [ stream atEnd] whileFalse: [ stream next. "..." ].
>>>
>>> is no longer allowed ?
>>
>> It hasn't been allowed "forever" [1].  It's just been misused for
>> almost as long.
>>
>> [1] Time began when stdio stream support was introduced. :-)
>
> I am still not convinced. Another way to put it would be that the old #atEnd 
> or #upToEnd do not make sense for these streams and some new loop is needed, 
> based on a new test (it exists for socket streams already).
>
> [ stream isDataAvailable ] whileTrue: [ stream next ]
>
>>> And you want to replace it with
>>>
>>> [ stream next ifNil: [ false ] ifNotNil: [ :x | "..." true ] whileTrue.
>>>
>>> That is a pretty big change, no ?
>>
>> That's the way quite a bit of code already operates.
>>
>> As Denis pointed out, it's obviously problematic in the general sense,
>> since nil can be embedded in non-byte oriented streams.  I suspect
>> that in practice not many people write code that reads streams from
>> both byte oriented and non-byte oriented streams.
>
> Maybe yes, maybe no. As Denis' example shows there is a clear definition 
> problem.
>
> And I do use streams of byte arrays or strings all the time, this is really 
> important. I want my parsers to work on all kinds of streams.
>
>>> I think/feel like a proper EOF exception would be better, more correct.
>>>
>>> [ [ stream next. "..." true ] on: EOF do: [ false ] ] whileTrue.
>>
>> I agree, but the email thread Nicolas pointed to raises some
>> performance questions about this approach.  It should be
>> straightforward to do a basic performance comparison which I'll get
>> around to if other objections aren't raised.
>
> Reading in bigger blocks, using #readInto:startingAt:count: (which is 
> basically Unix's (2) Read sys call), would solve performance problems, I 
> think.
>
>>> Will we throw away #atEnd then ? Do we need it if we cannot use it ?
>>
>> Unix file i/o returns EOF if the end of file has been reach OR if an
>> error occurs.  You should still check #atEnd after reading past the
>> end of the file to make sure no error occurred.  Another part of the
>> primitive change I'm proposing is to return additional information
>> about what went wrong in the event of an error.
>
> I am sorry, but this kind of semantics (the OR) is way too complex at the 
> general image level, it is too specific and based on certain underlying 
> implementation details.
>
> Sven
>
>> We could modify the read primitive so that it fails if an error has
>> occurred, and then #atEnd wouldn't be required.
>>
>> Cheers,
>> Alistair
>>
>>
>>
 On 4 Apr 2018, at 12:41, Alistair Grant  wrote:

 Hi Nicolas,

 On 4 April 2018 at 12:36, Nicolas Cellier
  wrote:
>
>
> 2018-04-04 12:18 GMT+02:00 Alistair Grant :
>>
>> Hi Sven,
>>
>> On Wed, Apr 04, 2018 at 11:32:02AM +0200, Sven Van Caekenberghe wrote:
>>> Somehow, somewhere there was a change to the implementation of the
>>> primitive called by some streams' #atEnd.
>>
>> That's a proposed change by me, but it hasn't been integrated yet.  So
>> the discussion below should apply to the current stable vm (from August
>> last year).
>>
>>
>>> IIRC, someone said it is implemented as 'remaining size being zero'
>>> and some virtual unix files like /dev/random are zero sized.
>>
>> Currently, for files other than sdio (stdout, stderr, stdin) it is
>> effectively defined as:
>>
>> atEnd := stream position >= stream size
>>
>>
>> And, as you say, plenty of virtual unix files report size 0.
>>
>>
>>
>>> Now, all kinds of changes are being done image size to work around this.
>>
>> I would phrase this slightly differently :-)
>>
>> Some code does the right thing, while other code doesn't.  E.g.:
>>
>> MultiByteFileStream>>upToEnd is good, while
>> FileStream>>contents is incorrect
>>
>>
>>> I am a strong believer in simple, real (i.e. infinite) streams, but I
>>> am not sure we are doing the right thing here.
>>>
>>> Point is, I am not sure #next returning nil is official and universal.
>>>
>>> Consider the comments:
>>>
>>> Stream>>#next
>>> "Answer the next object accessible by the receiver."
>>>
>>> ReadStream>>#next
>>> "Primitive. Answer the 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[K K Subbu]

On Wednesday 04 April 2018 09:07 PM, Sven Van Caekenberghe wrote:

Good summary, I agree.

Still, what are the semantics of #next - does the caller always have
to check for nil ? Do we think this is ugly (as the return value is
outside the domain) ? Do we then still need #atEnd ?


Senders of #next should be prepared to handle errors and EoS is just one 
of them.


On exhausting all elements in the collection, #next could block waiting 
for next object to become available or check for errors. Reading past 
End is just one of the error conditions. One way to recover from EoS 
error is to return an sentinel. The exact sentinel object could depend 
on the collection. Eliot suggested a good one earlier in this thread.


Regards .. Subbu

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Eliot Miranda]

On Wed, Apr 4, 2018 at 8:37 AM, Sven Van Caekenberghe  wrote:

>
>
> > On 4 Apr 2018, at 17:32, K K Subbu  wrote:
> >
> > On Wednesday 04 April 2018 04:06 PM, Nicolas Cellier wrote:
> >>> IIRC, someone said it is implemented as 'remaining size being zero'
> >>> and some virtual unix files like /dev/random are zero sized.
> >> Currently, for files other than sdio (stdout, stderr, stdin) it is
> >> effectively defined as:
> >> atEnd := stream position >= stream size
> > I see a confusion between Stream and its underlying collection. Stream
> is an iterator and just does next, nextPut, peek, reset etc. But methods
> like size or atEnd depend on its collection and there is no guarantee that
> this collection has a known and finite size.
> >
> > Essentially, a collection's size may be known and finite, unknown but
> finite size or infinite. This has nothing do with file descriptor being
> std{in,out,err}. If std* is a regular file or special file like /dev/mem,
> /dev/null, it's size is known at open time. With console streams or sysfs
> files, size is unknown until EOT (^D) or NUL is received. Lastly, special
> files like /dev/zero, /dev/random or /proc/cpuinfo don't have a finite size
> but report it as zero (!).
> >
> > [ stream atEnd ] whileFalse: [ stream next. .. ]
> >
> > will only terminate if its collection size is finite. It won't terminate
> for infinite collections.
> >
> > Regards .. Subbu
>
> Good summary, I agree.
>
> Still, what are the semantics of #next - does the caller always have to
> check for nil ? Do we think this is ugly (as the return value is outside
> the domain) ?


The problem is not that the value is outside the domain.  The problem is
that it may be within the domain.  Answering an element within the domain
when there are no more elements is insane.  Not even C does that.


> Do we then still need #atEnd ?
>

For backward compatibility, yes, even if deprecated.  There is so much
existing code that uses it, it'll be useful to have.


> Sven
>

_,,,^..^,,,_
best, Eliot

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

> On 4 Apr 2018, at 17:32, K K Subbu  wrote:
> 
> On Wednesday 04 April 2018 04:06 PM, Nicolas Cellier wrote:
>>> IIRC, someone said it is implemented as 'remaining size being zero'
>>> and some virtual unix files like /dev/random are zero sized.
>> Currently, for files other than sdio (stdout, stderr, stdin) it is
>> effectively defined as:
>> atEnd := stream position >= stream size
> I see a confusion between Stream and its underlying collection. Stream is an 
> iterator and just does next, nextPut, peek, reset etc. But methods like size 
> or atEnd depend on its collection and there is no guarantee that this 
> collection has a known and finite size.
> 
> Essentially, a collection's size may be known and finite, unknown but finite 
> size or infinite. This has nothing do with file descriptor being 
> std{in,out,err}. If std* is a regular file or special file like /dev/mem, 
> /dev/null, it's size is known at open time. With console streams or sysfs 
> files, size is unknown until EOT (^D) or NUL is received. Lastly, special 
> files like /dev/zero, /dev/random or /proc/cpuinfo don't have a finite size 
> but report it as zero (!).
> 
> [ stream atEnd ] whileFalse: [ stream next. .. ]
> 
> will only terminate if its collection size is finite. It won't terminate for 
> infinite collections.
> 
> Regards .. Subbu

Good summary, I agree.

Still, what are the semantics of #next - does the caller always have to check 
for nil ? Do we think this is ugly (as the return value is outside the domain) 
? Do we then still need #atEnd ?

Sven

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[K K Subbu]

On Wednesday 04 April 2018 04:06 PM, Nicolas Cellier wrote:

IIRC, someone said it is implemented as 'remaining size being zero'
and some virtual unix files like /dev/random are zero sized.


Currently, for files other than sdio (stdout, stderr, stdin) it is
effectively defined as:

atEnd := stream position >= stream size
I see a confusion between Stream and its underlying collection. Stream 
is an iterator and just does next, nextPut, peek, reset etc. But methods 
like size or atEnd depend on its collection and there is no guarantee 
that this collection has a known and finite size.


Essentially, a collection's size may be known and finite, unknown but 
finite size or infinite. This has nothing do with file descriptor being 
std{in,out,err}. If std* is a regular file or special file like 
/dev/mem, /dev/null, it's size is known at open time. With console 
streams or sysfs files, size is unknown until EOT (^D) or NUL is 
received. Lastly, special files like /dev/zero, /dev/random or 
/proc/cpuinfo don't have a finite size but report it as zero (!).


[ stream atEnd ] whileFalse: [ stream next. .. ]

will only terminate if its collection size is finite. It won't terminate 
for infinite collections.


Regards .. Subbu

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

Alistair,

First off, thanks for the discussions and your contributions, I really 
appreciate them.

But I want to have a discussion at the high level of the definition and 
semantics of the stream API in Pharo.

> On 4 Apr 2018, at 13:20, Alistair Grant  wrote:
> 
> On 4 April 2018 at 12:56, Sven Van Caekenberghe  wrote:
>> Playing a bit devil's advocate, the idea is that, in general,
>> 
>> [ stream atEnd] whileFalse: [ stream next. "..." ].
>> 
>> is no longer allowed ?
> 
> It hasn't been allowed "forever" [1].  It's just been misused for
> almost as long.
> 
> [1] Time began when stdio stream support was introduced. :-)

I am still not convinced. Another way to put it would be that the old #atEnd or 
#upToEnd do not make sense for these streams and some new loop is needed, based 
on a new test (it exists for socket streams already).

[ stream isDataAvailable ] whileTrue: [ stream next ]

>> And you want to replace it with
>> 
>> [ stream next ifNil: [ false ] ifNotNil: [ :x | "..." true ] whileTrue.
>> 
>> That is a pretty big change, no ?
> 
> That's the way quite a bit of code already operates.
> 
> As Denis pointed out, it's obviously problematic in the general sense,
> since nil can be embedded in non-byte oriented streams.  I suspect
> that in practice not many people write code that reads streams from
> both byte oriented and non-byte oriented streams.

Maybe yes, maybe no. As Denis' example shows there is a clear definition 
problem.

And I do use streams of byte arrays or strings all the time, this is really 
important. I want my parsers to work on all kinds of streams.

>> I think/feel like a proper EOF exception would be better, more correct.
>> 
>> [ [ stream next. "..." true ] on: EOF do: [ false ] ] whileTrue.
> 
> I agree, but the email thread Nicolas pointed to raises some
> performance questions about this approach.  It should be
> straightforward to do a basic performance comparison which I'll get
> around to if other objections aren't raised.

Reading in bigger blocks, using #readInto:startingAt:count: (which is basically 
Unix's (2) Read sys call), would solve performance problems, I think.

>> Will we throw away #atEnd then ? Do we need it if we cannot use it ?
> 
> Unix file i/o returns EOF if the end of file has been reach OR if an
> error occurs.  You should still check #atEnd after reading past the
> end of the file to make sure no error occurred.  Another part of the
> primitive change I'm proposing is to return additional information
> about what went wrong in the event of an error.

I am sorry, but this kind of semantics (the OR) is way too complex at the 
general image level, it is too specific and based on certain underlying 
implementation details.

Sven

> We could modify the read primitive so that it fails if an error has
> occurred, and then #atEnd wouldn't be required.
> 
> Cheers,
> Alistair
> 
> 
> 
>>> On 4 Apr 2018, at 12:41, Alistair Grant  wrote:
>>> 
>>> Hi Nicolas,
>>> 
>>> On 4 April 2018 at 12:36, Nicolas Cellier
>>>  wrote:
 
 
 2018-04-04 12:18 GMT+02:00 Alistair Grant :
> 
> Hi Sven,
> 
> On Wed, Apr 04, 2018 at 11:32:02AM +0200, Sven Van Caekenberghe wrote:
>> Somehow, somewhere there was a change to the implementation of the
>> primitive called by some streams' #atEnd.
> 
> That's a proposed change by me, but it hasn't been integrated yet.  So
> the discussion below should apply to the current stable vm (from August
> last year).
> 
> 
>> IIRC, someone said it is implemented as 'remaining size being zero'
>> and some virtual unix files like /dev/random are zero sized.
> 
> Currently, for files other than sdio (stdout, stderr, stdin) it is
> effectively defined as:
> 
> atEnd := stream position >= stream size
> 
> 
> And, as you say, plenty of virtual unix files report size 0.
> 
> 
> 
>> Now, all kinds of changes are being done image size to work around this.
> 
> I would phrase this slightly differently :-)
> 
> Some code does the right thing, while other code doesn't.  E.g.:
> 
> MultiByteFileStream>>upToEnd is good, while
> FileStream>>contents is incorrect
> 
> 
>> I am a strong believer in simple, real (i.e. infinite) streams, but I
>> am not sure we are doing the right thing here.
>> 
>> Point is, I am not sure #next returning nil is official and universal.
>> 
>> Consider the comments:
>> 
>> Stream>>#next
>> "Answer the next object accessible by the receiver."
>> 
>> ReadStream>>#next
>> "Primitive. Answer the next object in the Stream represented by the
>> receiver. Fail if the collection of this stream is not an Array or a
>> String.
>> Fail if the stream is positioned at its end, or if the position is out
>> of

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

On 4 April 2018 at 12:56, Sven Van Caekenberghe  wrote:
> Playing a bit devil's advocate, the idea is that, in general,
>
> [ stream atEnd] whileFalse: [ stream next. "..." ].
>
> is no longer allowed ?

It hasn't been allowed "forever" [1].  It's just been misused for
almost as long.

[1] Time began when stdio stream support was introduced. :-)


> And you want to replace it with
>
> [ stream next ifNil: [ false ] ifNotNil: [ :x | "..." true ] whileTrue.
>
> That is a pretty big change, no ?

That's the way quite a bit of code already operates.

As Denis pointed out, it's obviously problematic in the general sense,
since nil can be embedded in non-byte oriented streams.  I suspect
that in practice not many people write code that reads streams from
both byte oriented and non-byte oriented streams.


> I think/feel like a proper EOF exception would be better, more correct.
>
> [ [ stream next. "..." true ] on: EOF do: [ false ] ] whileTrue.

I agree, but the email thread Nicolas pointed to raises some
performance questions about this approach.  It should be
straightforward to do a basic performance comparison which I'll get
around to if other objections aren't raised.


> Will we throw away #atEnd then ? Do we need it if we cannot use it ?

Unix file i/o returns EOF if the end of file has been reach OR if an
error occurs.  You should still check #atEnd after reading past the
end of the file to make sure no error occurred.  Another part of the
primitive change I'm proposing is to return additional information
about what went wrong in the event of an error.

We could modify the read primitive so that it fails if an error has
occurred, and then #atEnd wouldn't be required.

Cheers,
Alistair



>> On 4 Apr 2018, at 12:41, Alistair Grant  wrote:
>>
>> Hi Nicolas,
>>
>> On 4 April 2018 at 12:36, Nicolas Cellier
>>  wrote:
>>>
>>>
>>> 2018-04-04 12:18 GMT+02:00 Alistair Grant :

 Hi Sven,

 On Wed, Apr 04, 2018 at 11:32:02AM +0200, Sven Van Caekenberghe wrote:
> Somehow, somewhere there was a change to the implementation of the
> primitive called by some streams' #atEnd.

 That's a proposed change by me, but it hasn't been integrated yet.  So
 the discussion below should apply to the current stable vm (from August
 last year).


> IIRC, someone said it is implemented as 'remaining size being zero'
> and some virtual unix files like /dev/random are zero sized.

 Currently, for files other than sdio (stdout, stderr, stdin) it is
 effectively defined as:

 atEnd := stream position >= stream size


 And, as you say, plenty of virtual unix files report size 0.



> Now, all kinds of changes are being done image size to work around this.

 I would phrase this slightly differently :-)

 Some code does the right thing, while other code doesn't.  E.g.:

 MultiByteFileStream>>upToEnd is good, while
 FileStream>>contents is incorrect


> I am a strong believer in simple, real (i.e. infinite) streams, but I
> am not sure we are doing the right thing here.
>
> Point is, I am not sure #next returning nil is official and universal.
>
> Consider the comments:
>
> Stream>>#next
>  "Answer the next object accessible by the receiver."
>
> ReadStream>>#next
>  "Primitive. Answer the next object in the Stream represented by the
>  receiver. Fail if the collection of this stream is not an Array or a
> String.
>  Fail if the stream is positioned at its end, or if the position is out
> of
>  bounds in the collection. Optional. See Object documentation
>  whatIsAPrimitive."
>
> Note how there is no talk about returning nil !
>
> I think we should discuss about this first.
>
> Was the low level change really correct and the right thing to do ?

 The primitive change proposed doesn't affect this discussion.  It will
 mean that #atEnd returns false (correctly) sometimes, while currently it
 returns true (incorrectly).  The end result is still incorrect, e.g.
 #contents returns an empty string for /proc/cpuinfo.

 You're correct about no mention of nil, but we have:

 FileStream>>next

(position >= readLimit and: [self atEnd])
ifTrue: [^nil]
ifFalse: [^collection at: (position := position + 1)]


 which has been around for a long time (I suspect, before Pharo existed).

 Having said that, I think that raising an exception is a better
 solution, but it is a much, much bigger change than the one I proposed
 in https://github.com/pharo-project/pharo/pull/1180.


 Cheers,
 Alistair

>>>
>>> Hi,
>>> yes, if you are after universal behavior englobing Unix streams, the
>>> Exception 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

Playing a bit devil's advocate, the idea is that, in general, 

[ stream atEnd] whileFalse: [ stream next. "..." ].

is no longer allowed ? And you want to replace it with

[ stream next ifNil: [ false ] ifNotNil: [ :x | "..." true ] whileTrue.

That is a pretty big change, no ?

I think/feel like a proper EOF exception would be better, more correct.

[ [ stream next. "..." true ] on: EOF do: [ false ] ] whileTrue.

Will we throw away #atEnd then ? Do we need it if we cannot use it ?

> On 4 Apr 2018, at 12:41, Alistair Grant  wrote:
> 
> Hi Nicolas,
> 
> On 4 April 2018 at 12:36, Nicolas Cellier
>  wrote:
>> 
>> 
>> 2018-04-04 12:18 GMT+02:00 Alistair Grant :
>>> 
>>> Hi Sven,
>>> 
>>> On Wed, Apr 04, 2018 at 11:32:02AM +0200, Sven Van Caekenberghe wrote:
 Somehow, somewhere there was a change to the implementation of the
 primitive called by some streams' #atEnd.
>>> 
>>> That's a proposed change by me, but it hasn't been integrated yet.  So
>>> the discussion below should apply to the current stable vm (from August
>>> last year).
>>> 
>>> 
 IIRC, someone said it is implemented as 'remaining size being zero'
 and some virtual unix files like /dev/random are zero sized.
>>> 
>>> Currently, for files other than sdio (stdout, stderr, stdin) it is
>>> effectively defined as:
>>> 
>>> atEnd := stream position >= stream size
>>> 
>>> 
>>> And, as you say, plenty of virtual unix files report size 0.
>>> 
>>> 
>>> 
 Now, all kinds of changes are being done image size to work around this.
>>> 
>>> I would phrase this slightly differently :-)
>>> 
>>> Some code does the right thing, while other code doesn't.  E.g.:
>>> 
>>> MultiByteFileStream>>upToEnd is good, while
>>> FileStream>>contents is incorrect
>>> 
>>> 
 I am a strong believer in simple, real (i.e. infinite) streams, but I
 am not sure we are doing the right thing here.
 
 Point is, I am not sure #next returning nil is official and universal.
 
 Consider the comments:
 
 Stream>>#next
  "Answer the next object accessible by the receiver."
 
 ReadStream>>#next
  "Primitive. Answer the next object in the Stream represented by the
  receiver. Fail if the collection of this stream is not an Array or a
 String.
  Fail if the stream is positioned at its end, or if the position is out
 of
  bounds in the collection. Optional. See Object documentation
  whatIsAPrimitive."
 
 Note how there is no talk about returning nil !
 
 I think we should discuss about this first.
 
 Was the low level change really correct and the right thing to do ?
>>> 
>>> The primitive change proposed doesn't affect this discussion.  It will
>>> mean that #atEnd returns false (correctly) sometimes, while currently it
>>> returns true (incorrectly).  The end result is still incorrect, e.g.
>>> #contents returns an empty string for /proc/cpuinfo.
>>> 
>>> You're correct about no mention of nil, but we have:
>>> 
>>> FileStream>>next
>>> 
>>>(position >= readLimit and: [self atEnd])
>>>ifTrue: [^nil]
>>>ifFalse: [^collection at: (position := position + 1)]
>>> 
>>> 
>>> which has been around for a long time (I suspect, before Pharo existed).
>>> 
>>> Having said that, I think that raising an exception is a better
>>> solution, but it is a much, much bigger change than the one I proposed
>>> in https://github.com/pharo-project/pharo/pull/1180.
>>> 
>>> 
>>> Cheers,
>>> Alistair
>>> 
>> 
>> Hi,
>> yes, if you are after universal behavior englobing Unix streams, the
>> Exception might be the best way.
>> Because on special stream you can't allways say in advance, you have to try.
>> That's the solution adopted by authors of Xtreams.
>> But there is a runtime penalty associated to it.
>> 
>> The penalty once was so high that my proposal to generalize EndOfStream
>> usage was rejected a few years ago by AndreaRaab.
>> http://forum.world.st/EndOfStream-unused-td68806.html
> 
> Thanks for this, I'll definitely take a look.
> 
> Do you have a sense of how Denis' suggestion of using an EndOfStream
> object would compare?
> 
> It would keep the same coding style, but avoid the problems with nil.
> 
> Thanks,
> Alistair
> 
> 
> 
>> I have regularly benched Xtreams, but stopped a few years ago.
>> Maybe i can excavate and pass on newer VM.
>> 
>> In the mean time, i had experimented a programmable end of stream behavior
>> (via a block, or any other valuable)
>> http://www.squeaksource.com/XTream.htm
>> so as to reconcile performance and universality, but it was a source of
>> complexification at implementation side.
>> 
>> Nicolas
>> 
>>> 
>>> 
 Note also that a Guille introduced something new, #closed which is
 related to the difference between having no more elements (maybe right now,
 like an open network stream) and never ever being able to produce 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

Hi Nicolas,

On 4 April 2018 at 12:36, Nicolas Cellier
 wrote:
>
>
> 2018-04-04 12:18 GMT+02:00 Alistair Grant :
>>
>> Hi Sven,
>>
>> On Wed, Apr 04, 2018 at 11:32:02AM +0200, Sven Van Caekenberghe wrote:
>> > Somehow, somewhere there was a change to the implementation of the
>> > primitive called by some streams' #atEnd.
>>
>> That's a proposed change by me, but it hasn't been integrated yet.  So
>> the discussion below should apply to the current stable vm (from August
>> last year).
>>
>>
>> > IIRC, someone said it is implemented as 'remaining size being zero'
>> > and some virtual unix files like /dev/random are zero sized.
>>
>> Currently, for files other than sdio (stdout, stderr, stdin) it is
>> effectively defined as:
>>
>> atEnd := stream position >= stream size
>>
>>
>> And, as you say, plenty of virtual unix files report size 0.
>>
>>
>>
>> > Now, all kinds of changes are being done image size to work around this.
>>
>> I would phrase this slightly differently :-)
>>
>> Some code does the right thing, while other code doesn't.  E.g.:
>>
>> MultiByteFileStream>>upToEnd is good, while
>> FileStream>>contents is incorrect
>>
>>
>> > I am a strong believer in simple, real (i.e. infinite) streams, but I
>> > am not sure we are doing the right thing here.
>> >
>> > Point is, I am not sure #next returning nil is official and universal.
>> >
>> > Consider the comments:
>> >
>> > Stream>>#next
>> >   "Answer the next object accessible by the receiver."
>> >
>> > ReadStream>>#next
>> >   "Primitive. Answer the next object in the Stream represented by the
>> >   receiver. Fail if the collection of this stream is not an Array or a
>> > String.
>> >   Fail if the stream is positioned at its end, or if the position is out
>> > of
>> >   bounds in the collection. Optional. See Object documentation
>> >   whatIsAPrimitive."
>> >
>> > Note how there is no talk about returning nil !
>> >
>> > I think we should discuss about this first.
>> >
>> > Was the low level change really correct and the right thing to do ?
>>
>> The primitive change proposed doesn't affect this discussion.  It will
>> mean that #atEnd returns false (correctly) sometimes, while currently it
>> returns true (incorrectly).  The end result is still incorrect, e.g.
>> #contents returns an empty string for /proc/cpuinfo.
>>
>> You're correct about no mention of nil, but we have:
>>
>> FileStream>>next
>>
>> (position >= readLimit and: [self atEnd])
>> ifTrue: [^nil]
>> ifFalse: [^collection at: (position := position + 1)]
>>
>>
>> which has been around for a long time (I suspect, before Pharo existed).
>>
>> Having said that, I think that raising an exception is a better
>> solution, but it is a much, much bigger change than the one I proposed
>> in https://github.com/pharo-project/pharo/pull/1180.
>>
>>
>> Cheers,
>> Alistair
>>
>
> Hi,
> yes, if you are after universal behavior englobing Unix streams, the
> Exception might be the best way.
> Because on special stream you can't allways say in advance, you have to try.
> That's the solution adopted by authors of Xtreams.
> But there is a runtime penalty associated to it.
>
> The penalty once was so high that my proposal to generalize EndOfStream
> usage was rejected a few years ago by AndreaRaab.
> http://forum.world.st/EndOfStream-unused-td68806.html

Thanks for this, I'll definitely take a look.

Do you have a sense of how Denis' suggestion of using an EndOfStream
object would compare?

It would keep the same coding style, but avoid the problems with nil.

Thanks,
Alistair



> I have regularly benched Xtreams, but stopped a few years ago.
> Maybe i can excavate and pass on newer VM.
>
> In the mean time, i had experimented a programmable end of stream behavior
> (via a block, or any other valuable)
> http://www.squeaksource.com/XTream.htm
> so as to reconcile performance and universality, but it was a source of
> complexification at implementation side.
>
> Nicolas
>
>>
>>
>> > Note also that a Guille introduced something new, #closed which is
>> > related to the difference between having no more elements (maybe right now,
>> > like an open network stream) and never ever being able to produce more 
>> > data.
>> >
>> > Sven
>> >
>> >
>> >
>>
>

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Nicolas Cellier]

2018-04-04 12:18 GMT+02:00 Alistair Grant :

> Hi Sven,
>
> On Wed, Apr 04, 2018 at 11:32:02AM +0200, Sven Van Caekenberghe wrote:
> > Somehow, somewhere there was a change to the implementation of the
> > primitive called by some streams' #atEnd.
>
> That's a proposed change by me, but it hasn't been integrated yet.  So
> the discussion below should apply to the current stable vm (from August
> last year).
>
>
> > IIRC, someone said it is implemented as 'remaining size being zero'
> > and some virtual unix files like /dev/random are zero sized.
>
> Currently, for files other than sdio (stdout, stderr, stdin) it is
> effectively defined as:
>
> atEnd := stream position >= stream size
>
>
> And, as you say, plenty of virtual unix files report size 0.
>
>
>
> > Now, all kinds of changes are being done image size to work around this.
>
> I would phrase this slightly differently :-)
>
> Some code does the right thing, while other code doesn't.  E.g.:
>
> MultiByteFileStream>>upToEnd is good, while
> FileStream>>contents is incorrect
>
>
> > I am a strong believer in simple, real (i.e. infinite) streams, but I
> > am not sure we are doing the right thing here.
> >
> > Point is, I am not sure #next returning nil is official and universal.
> >
> > Consider the comments:
> >
> > Stream>>#next
> >   "Answer the next object accessible by the receiver."
> >
> > ReadStream>>#next
> >   "Primitive. Answer the next object in the Stream represented by the
> >   receiver. Fail if the collection of this stream is not an Array or a
> String.
> >   Fail if the stream is positioned at its end, or if the position is out
> of
> >   bounds in the collection. Optional. See Object documentation
> >   whatIsAPrimitive."
> >
> > Note how there is no talk about returning nil !
> >
> > I think we should discuss about this first.
> >
> > Was the low level change really correct and the right thing to do ?
>
> The primitive change proposed doesn't affect this discussion.  It will
> mean that #atEnd returns false (correctly) sometimes, while currently it
> returns true (incorrectly).  The end result is still incorrect, e.g.
> #contents returns an empty string for /proc/cpuinfo.
>
> You're correct about no mention of nil, but we have:
>
> FileStream>>next
>
> (position >= readLimit and: [self atEnd])
> ifTrue: [^nil]
> ifFalse: [^collection at: (position := position + 1)]
>
>
> which has been around for a long time (I suspect, before Pharo existed).
>
> Having said that, I think that raising an exception is a better
> solution, but it is a much, much bigger change than the one I proposed
> in https://github.com/pharo-project/pharo/pull/1180.
>
>
> Cheers,
> Alistair
>
>
Hi,
yes, if you are after universal behavior englobing Unix streams, the
Exception might be the best way.
Because on special stream you can't allways say in advance, you have to try.
That's the solution adopted by authors of Xtreams.
But there is a runtime penalty associated to it.

The penalty once was so high that my proposal to generalize EndOfStream
usage was rejected a few years ago by AndreaRaab.
http://forum.world.st/EndOfStream-unused-td68806.html

I have regularly benched Xtreams, but stopped a few years ago.
Maybe i can excavate and pass on newer VM.

In the mean time, i had experimented a programmable end of stream behavior
(via a block, or any other valuable)
http://www.squeaksource.com/XTream.htm
so as to reconcile performance and universality, but it was a source of
complexification at implementation side.

Nicolas


>
> > Note also that a Guille introduced something new, #closed which is
> related to the difference between having no more elements (maybe right now,
> like an open network stream) and never ever being able to produce more data.
> >
> > Sven
> >
> >
> >
>
>

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Alistair Grant]

Hi Sven,

On Wed, Apr 04, 2018 at 11:32:02AM +0200, Sven Van Caekenberghe wrote:
> Somehow, somewhere there was a change to the implementation of the
> primitive called by some streams' #atEnd. 

That's a proposed change by me, but it hasn't been integrated yet.  So
the discussion below should apply to the current stable vm (from August
last year).


> IIRC, someone said it is implemented as 'remaining size being zero'
> and some virtual unix files like /dev/random are zero sized.

Currently, for files other than sdio (stdout, stderr, stdin) it is
effectively defined as:

atEnd := stream position >= stream size


And, as you say, plenty of virtual unix files report size 0.



> Now, all kinds of changes are being done image size to work around this.

I would phrase this slightly differently :-)

Some code does the right thing, while other code doesn't.  E.g.:

MultiByteFileStream>>upToEnd is good, while
FileStream>>contents is incorrect


> I am a strong believer in simple, real (i.e. infinite) streams, but I
> am not sure we are doing the right thing here.
> 
> Point is, I am not sure #next returning nil is official and universal.
> 
> Consider the comments:
> 
> Stream>>#next
>   "Answer the next object accessible by the receiver."
> 
> ReadStream>>#next
>   "Primitive. Answer the next object in the Stream represented by the
>   receiver. Fail if the collection of this stream is not an Array or a String.
>   Fail if the stream is positioned at its end, or if the position is out of
>   bounds in the collection. Optional. See Object documentation
>   whatIsAPrimitive."
> 
> Note how there is no talk about returning nil !
> 
> I think we should discuss about this first.
> 
> Was the low level change really correct and the right thing to do ?

The primitive change proposed doesn't affect this discussion.  It will
mean that #atEnd returns false (correctly) sometimes, while currently it
returns true (incorrectly).  The end result is still incorrect, e.g.
#contents returns an empty string for /proc/cpuinfo.

You're correct about no mention of nil, but we have:

FileStream>>next

(position >= readLimit and: [self atEnd])
ifTrue: [^nil]
ifFalse: [^collection at: (position := position + 1)]


which has been around for a long time (I suspect, before Pharo existed).

Having said that, I think that raising an exception is a better
solution, but it is a much, much bigger change than the one I proposed
in https://github.com/pharo-project/pharo/pull/1180.


Cheers,
Alistair



> Note also that a Guille introduced something new, #closed which is related to 
> the difference between having no more elements (maybe right now, like an open 
> network stream) and never ever being able to produce more data.
> 
> Sven
> 
> 
> 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Denis Kudriashov]

Hi.

#next returning nil is definitely looks bad because we will always have
case when nil value will not means the end:

#(1 nil 2) readSteam next; next; atEnd

But I understand that in many cases analysing #next for nil is much more
suitable than #atEnd. I am sure that it can be always avoided but it can be
not easy to do.
Alternatively we can reify real EndOfStream object which will be used as
result of reading operation when stream is at end.
It can provide nice API like #ifEndOfStream:. which will reflect intention
in much cleaner way than ifNil: checks.


2018-04-04 12:00 GMT+02:00 Sven Van Caekenberghe :

>
>
> > On 4 Apr 2018, at 11:38, Nicolas Cellier  gmail.com> wrote:
> >
> > Hi Sven,
> > See also discussion at https://github.com/OpenSmalltalk/opensmalltalk-
> vm/pull/232
>
> Thanks Nicolas, I had already seen parts of it.
>
> Now, I still want image level changes to be based on clear semantic
> definitions of the abstract stream API, we're not just talking about file
> or unix streams, the general Smalltalk concept.
>
> For reading, they are IMHO,
>
> #next
> #readInto:startingAt:count:
> #peek
> #atEnd
> #upToEnd (can be derived but still the semantics are important in relation
> to #atEnd)
>
> For writing, we have
>
> #nextPut:
> #next:putAll:startingAt:
> #flush
>
> For both, we have
>
> #atEnd
> #close
> #closed (new)
>
> So, I know, #next returning nil exists, but is it universally/officially
> defined as such ? Where is that documented ?
>
> Positioning, sizing are not universal, IMHO, but should be clearly defined
> as well.
>
> > 2018-04-04 11:32 GMT+02:00 Sven Van Caekenberghe :
> > Somehow, somewhere there was a change to the implementation of the
> primitive called by some streams' #atEnd.
> >
> > IIRC, someone said it is implemented as 'remaining size being zero' and
> some virtual unix files like /dev/random are zero sized.
> >
> > Now, all kinds of changes are being done image size to work around this.
> >
> > I am a strong believer in simple, real (i.e. infinite) streams, but I am
> not sure we are doing the right thing here.
> >
> > Point is, I am not sure #next returning nil is official and universal.
> >
> > Consider the comments:
> >
> > Stream>>#next
> >   "Answer the next object accessible by the receiver."
> >
> > ReadStream>>#next
> >   "Primitive. Answer the next object in the Stream represented by the
> >   receiver. Fail if the collection of this stream is not an Array or a
> String.
> >   Fail if the stream is positioned at its end, or if the position is out
> of
> >   bounds in the collection. Optional. See Object documentation
> >   whatIsAPrimitive."
> >
> > Note how there is no talk about returning nil !
> >
> > I think we should discuss about this first.
> >
> > Was the low level change really correct and the right thing to do ?
> >
> > Note also that a Guille introduced something new, #closed which is
> related to the difference between having no more elements (maybe right now,
> like an open network stream) and never ever being able to produce more data.
> >
> > Sven
> >
> >
> >
> >
>
>
>

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Sven Van Caekenberghe]

> On 4 Apr 2018, at 11:38, Nicolas Cellier  
> wrote:
> 
> Hi Sven,
> See also discussion at 
> https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/232

Thanks Nicolas, I had already seen parts of it.

Now, I still want image level changes to be based on clear semantic definitions 
of the abstract stream API, we're not just talking about file or unix streams, 
the general Smalltalk concept.

For reading, they are IMHO,

#next
#readInto:startingAt:count:
#peek
#atEnd
#upToEnd (can be derived but still the semantics are important in relation to 
#atEnd)

For writing, we have

#nextPut:
#next:putAll:startingAt: 
#flush

For both, we have

#atEnd
#close
#closed (new)

So, I know, #next returning nil exists, but is it universally/officially 
defined as such ? Where is that documented ?

Positioning, sizing are not universal, IMHO, but should be clearly defined as 
well.

> 2018-04-04 11:32 GMT+02:00 Sven Van Caekenberghe :
> Somehow, somewhere there was a change to the implementation of the primitive 
> called by some streams' #atEnd.
> 
> IIRC, someone said it is implemented as 'remaining size being zero' and some 
> virtual unix files like /dev/random are zero sized.
> 
> Now, all kinds of changes are being done image size to work around this.
> 
> I am a strong believer in simple, real (i.e. infinite) streams, but I am not 
> sure we are doing the right thing here.
> 
> Point is, I am not sure #next returning nil is official and universal.
> 
> Consider the comments:
> 
> Stream>>#next
>   "Answer the next object accessible by the receiver."
> 
> ReadStream>>#next
>   "Primitive. Answer the next object in the Stream represented by the
>   receiver. Fail if the collection of this stream is not an Array or a String.
>   Fail if the stream is positioned at its end, or if the position is out of
>   bounds in the collection. Optional. See Object documentation
>   whatIsAPrimitive."
> 
> Note how there is no talk about returning nil !
> 
> I think we should discuss about this first.
> 
> Was the low level change really correct and the right thing to do ?
> 
> Note also that a Guille introduced something new, #closed which is related to 
> the difference between having no more elements (maybe right now, like an open 
> network stream) and never ever being able to produce more data.
> 
> Sven
> 
> 
> 
> 

Re: [Pharo-dev] Changed #atEnd primitive - #atEnd vs #next returning nil

[Nicolas Cellier]

Hi Sven,
See also discussion at
https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/232

2018-04-04 11:32 GMT+02:00 Sven Van Caekenberghe :

> Somehow, somewhere there was a change to the implementation of the
> primitive called by some streams' #atEnd.
>
> IIRC, someone said it is implemented as 'remaining size being zero' and
> some virtual unix files like /dev/random are zero sized.
>
> Now, all kinds of changes are being done image size to work around this.
>
> I am a strong believer in simple, real (i.e. infinite) streams, but I am
> not sure we are doing the right thing here.
>
> Point is, I am not sure #next returning nil is official and universal.
>
> Consider the comments:
>
> Stream>>#next
>   "Answer the next object accessible by the receiver."
>
> ReadStream>>#next
>   "Primitive. Answer the next object in the Stream represented by the
>   receiver. Fail if the collection of this stream is not an Array or a
> String.
>   Fail if the stream is positioned at its end, or if the position is out of
>   bounds in the collection. Optional. See Object documentation
>   whatIsAPrimitive."
>
> Note how there is no talk about returning nil !
>
> I think we should discuss about this first.
>
> Was the low level change really correct and the right thing to do ?
>
> Note also that a Guille introduced something new, #closed which is related
> to the difference between having no more elements (maybe right now, like an
> open network stream) and never ever being able to produce more data.
>
> Sven
>
>
>
>