Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Ben Coman]

On Wed, Dec 23, 2015 at 7:46 PM, p...@highoctane.be  wrote:
>
> Pharo follows Smalltalk which uses a global object named Smalltalk which in
> fact is an instance of SmalltalkImage, confusing if you ask me.

I also have felt that same impedance mismatch in the naming, but I
thought the accessors were deliberately moved to the instance side to
help (maybe later) to minimise global state.  So I am guessing that
adding class-side accessors is a step backwards - but maybe System
class could provide a singleton instance of itself.

cheers -ben

>
> I would add a simple class named System which have class-side accessors for
> static global things like
>
> image => links to the old Smalltalk
> stdin
> stdout
> session
> environment for ENV vars
> workingDirectory accessing the process working directory $PWD
> 'pid'
>
> That's a very easy thing to do, but IMO it will help anybody coming from
> outside smalltalk to easily find these things. Probably there are more
> things that would fit here.
>
>
> I'd like to have that.
>
> The discussion involved Camillo who was very into these topics. His
> Pinocchio work seemed to have contributions that can be useful of the CLI
> support for Pharo.
>
> Phil
>
>>
>>
>>>
>>> and just btw, mildly interesting is using sockets rather than pipes
>>> for interprocess communication [l]
>>>
>>> [j]
>>> http://stackoverflow.com/questions/3884103/can-popen-make-bidirectional-pipes-like-pipe-fork
>>> [k] http://stackoverflow.com/questions/6743771/popen-alternative
>>> [l]
>>> http://stackoverflow.com/questions/25161377/open-a-cmd-program-with-full-functionality-i-o/25177958#25177958
>>>
>>
>> Thanks. I will check that on a second step ;)
>>
>>>
>>> >
>>> > 2) We believe that there is a large number of OSProcess users that use
>>> > OSProcess only for executing OS commands AND popen() approach would be
>>> > more
>>> > than enough (we will do a survey about this). So, a very simply
>>> > solution via
>>> > FFI for these users rather than having all OSProcess, Command Shell,
>>> > the
>>> > plugins etc, may be worth.
>>> >
>>> > What do you think?
>>> >
>>> >>
>>> >> Now, given some of the difficulties seen when dealing with external
>>> >> processes, I'd be wary of the "async" side of things. Random failures
>>> >> seemingly linked to signal loss are certainly not something I'd like
>>> >> to
>>> >> debug.
>>> >>
>>> >
>>> > Indeed, it won't be easy I guess. But sometimes the process you execute
>>> > may
>>> > take quite some time and blocking the VM for that period is not a good
>>> > choice. So... what would you prefer to solve that? Do you prefer
>>> > image-based
>>> > pooling as OSProcess does when IOPlugin is not used?
>>>
>>>
>>> How much might your work overlap with the Threaded FFI project for
>>> Cog?...
>>> ..."The Cog VM is single-threaded, providing green threads to the
>>> Smalltalk level, as do most Smalltalk VMs. There is a prototype
>>> subclass of the CoInterpreter, CoInterpreterMT, which provides a
>>> Python-like multi-threaded VM where any number of threads can share
>>> the VM with only one running the VM at any one time.  This uses an
>>> extremely simple low-level scheme for releasing and re-acquiring
>>> ownership of the VM designed by David Simmons.  This results in any
>>> and all FFI calls being non-blocking since there is such a low
>>> overhead to making a non-blocking FFI call that all FFI calls can be
>>> non-blocking.  Eliot has worked on this, implementing the prototype
>>> and the reentrant FFI plugin it requires.  But this needs to be
>>> revived.  It would be a really powerful enhancement."
>>>
>>
>> Well, Eliot said I may need to give a hand to them in order to finish the
>> Threaded FFI. However, I am still not 100% certain if the threaded FFI calls
>> to read from pipes would be worth. Eliot said that yes. Anyway, once I have
>> the first version working I will digest this and goes to the next level and
>> see how I can make things better than polling, if polling proves to be a
>> problem.
>>
>>
>>>
>>> cheers -ben
>>>
>>> P.S. Just another article warning against mixing fork and
>>> multiple-threads (which it seems posix_spawn may avoid)...
>>>
>>> http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them
>>>
>>
>> Thanks for the reference.
>>
>> Cheers,
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com
>
>

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[p...@highoctane.be]

On Wed, Dec 23, 2015 at 1:54 PM, Mariano Martinez Peck <
marianop...@gmail.com> wrote:

>
>>>
>>>
 Also I read [j] that a problem with popen is that you do not get pid
 of child so you cannot wait on command to complete.  Could posix_spawn
 be preferrable to popen?


>>> Exactly. There is no standard way to get child pid from popen() so you
>>> can't do polling in image side to wait it's completion.
>>> So yeah, posix_spawn() does fix this issue as well as many many other.
>>>
>>
>>
>> Looks nice and at the same time, popen support is useful/wanted.
>>
>
> Yes, but I wonder if it would still make sense if I finish the posix_spawn
> flavor. Anyway, we can decide later since the Popen wrapper is almost
> written (a very very first prototype)
>
>
>> A generic way to deal with pids (in Linux at least, is to have the
>> process write its pid in /var/run/ or a subdir of it.
>>
>
> That's right, but that only works for a certain number of cases, and
> polling with checking files does not seem good.
>

Lots of software already works that way. In the name of proper integration
of external processes/"things", it is a must have.
Well, I need it in production.


> I think that you if you expect to run a a command that takes some time,
> then use the posix_spawn wrapper and use popen only in "blocking" mode.
>
>
>>
>> That's how monit checks for processes for example.
>>
>> http://unix.stackexchange.com/questions/12815/what-are-pid-and-lock-files-for
>>
>> We need as many options as we can here.
>>
>
> I not fully convinced with that statement.
>

Anything making Pharo less of an island is a step in the right direction.
popen is how it is done in one of the most successful platforms. I am using
it too, so, being able to port knowledge from point A to B is useful in my
view. https://docs.python.org/2/library/subprocess.html You'll see spawn
and popen. And Popen2,3, 4.


>
>
>> Mariano, I added you to a Trello board that captured a couple of items
>> about the topic you deal with.
>> https://trello.com/b/Hji4F0Zp/clipharo
>>
>>
> Thanks!
>
>
>> We also have a card that says the following:
>>
>> Pharo follows Smalltalk which uses a global object named Smalltalk which
>> in fact is an instance of SmalltalkImage, confusing if you ask me.
>>
>> I would add a simple class named System which have class-side accessors
>> for static global things like
>>
>
> My answers below. Just as a matter of how that System class could be
> written:
>
>
>>
>>- image => links to the old Smalltalk
>>
>>
> Well..we have now "Smalltalk image"
>
>
>>- stdin
>>- stdout
>>- session
>>
>>
> OK about those.
>
>
>>
>>- environment for ENV vars
>>
>>
> we have "Smalltalk os environment"
>
>
>>
>>- workingDirectory accessing the process working directory $PWD
>>
>>
> We have "FileSystem workingDirectory"
>

Look above, one has to peek all around to find them (not to mention being
aware. And I'd even say, foreget FileSystem, why not FileLocator, which is
meant for that).

FWIW, https://nodejs.org/api/os.html


>
>
>>
>>- 'pid'
>>
>>
> Well, we need OSProcess right now, but it is also very very easy to do via
> FFI.
>
>
>
>> That's a very easy thing to do, but IMO it will help anybody coming from
>> outside smalltalk to easily find these things. Probably there are more
>> things that would fit here.
>>
>> I'd like to have that.
>>
>> The discussion involved Camillo who was very into these topics. His
>> Pinocchio work seemed to have contributions that can be useful of the CLI
>> support for Pharo.
>>
>> Phil
>>
>>
>>>
>>>
 and just btw, mildly interesting is using sockets rather than pipes
 for interprocess communication [l]

 [j]
 http://stackoverflow.com/questions/3884103/can-popen-make-bidirectional-pipes-like-pipe-fork
 [k] http://stackoverflow.com/questions/6743771/popen-alternative
 [l]
 http://stackoverflow.com/questions/25161377/open-a-cmd-program-with-full-functionality-i-o/25177958#25177958


>>> Thanks. I will check that on a second step ;)
>>>
>>>
 >
 > 2) We believe that there is a large number of OSProcess users that use
 > OSProcess only for executing OS commands AND popen() approach would
 be more
 > than enough (we will do a survey about this). So, a very simply
 solution via
 > FFI for these users rather than having all OSProcess, Command Shell,
 the
 > plugins etc, may be worth.
 >
 > What do you think?
 >
 >>
 >> Now, given some of the difficulties seen when dealing with external
 >> processes, I'd be wary of the "async" side of things. Random failures
 >> seemingly linked to signal loss are certainly not something I'd like
 to
 >> debug.
 >>
 >
 > Indeed, it won't be easy I guess. But sometimes the process you
 execute may
 > take quite some time and blocking the VM for that period is not a good
 > choice. So... what would you prefer to solve that? Do 

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Mariano Martinez Peck]

>
>
>>
>>
>>> Also I read [j] that a problem with popen is that you do not get pid
>>> of child so you cannot wait on command to complete.  Could posix_spawn
>>> be preferrable to popen?
>>>
>>>
>> Exactly. There is no standard way to get child pid from popen() so you
>> can't do polling in image side to wait it's completion.
>> So yeah, posix_spawn() does fix this issue as well as many many other.
>>
>
>
> Looks nice and at the same time, popen support is useful/wanted.
>

Yes, but I wonder if it would still make sense if I finish the posix_spawn
flavor. Anyway, we can decide later since the Popen wrapper is almost
written (a very very first prototype)


> A generic way to deal with pids (in Linux at least, is to have the process
> write its pid in /var/run/ or a subdir of it.
>

That's right, but that only works for a certain number of cases, and
polling with checking files does not seem good.
I think that you if you expect to run a a command that takes some time,
then use the posix_spawn wrapper and use popen only in "blocking" mode.


>
> That's how monit checks for processes for example.
>
> http://unix.stackexchange.com/questions/12815/what-are-pid-and-lock-files-for
>
> We need as many options as we can here.
>

I not fully convinced with that statement.


> Mariano, I added you to a Trello board that captured a couple of items
> about the topic you deal with.
> https://trello.com/b/Hji4F0Zp/clipharo
>
>
Thanks!


> We also have a card that says the following:
>
> Pharo follows Smalltalk which uses a global object named Smalltalk which
> in fact is an instance of SmalltalkImage, confusing if you ask me.
>
> I would add a simple class named System which have class-side accessors
> for static global things like
>

My answers below. Just as a matter of how that System class could be
written:


>
>- image => links to the old Smalltalk
>
>
Well..we have now "Smalltalk image"


>- stdin
>- stdout
>- session
>
>
OK about those.


>
>- environment for ENV vars
>
>
we have "Smalltalk os environment"


>
>- workingDirectory accessing the process working directory $PWD
>
>
We have "FileSystem workingDirectory"


>
>- 'pid'
>
>
Well, we need OSProcess right now, but it is also very very easy to do via
FFI.



> That's a very easy thing to do, but IMO it will help anybody coming from
> outside smalltalk to easily find these things. Probably there are more
> things that would fit here.
>
> I'd like to have that.
>
> The discussion involved Camillo who was very into these topics. His
> Pinocchio work seemed to have contributions that can be useful of the CLI
> support for Pharo.
>
> Phil
>
>
>>
>>
>>> and just btw, mildly interesting is using sockets rather than pipes
>>> for interprocess communication [l]
>>>
>>> [j]
>>> http://stackoverflow.com/questions/3884103/can-popen-make-bidirectional-pipes-like-pipe-fork
>>> [k] http://stackoverflow.com/questions/6743771/popen-alternative
>>> [l]
>>> http://stackoverflow.com/questions/25161377/open-a-cmd-program-with-full-functionality-i-o/25177958#25177958
>>>
>>>
>> Thanks. I will check that on a second step ;)
>>
>>
>>> >
>>> > 2) We believe that there is a large number of OSProcess users that use
>>> > OSProcess only for executing OS commands AND popen() approach would be
>>> more
>>> > than enough (we will do a survey about this). So, a very simply
>>> solution via
>>> > FFI for these users rather than having all OSProcess, Command Shell,
>>> the
>>> > plugins etc, may be worth.
>>> >
>>> > What do you think?
>>> >
>>> >>
>>> >> Now, given some of the difficulties seen when dealing with external
>>> >> processes, I'd be wary of the "async" side of things. Random failures
>>> >> seemingly linked to signal loss are certainly not something I'd like
>>> to
>>> >> debug.
>>> >>
>>> >
>>> > Indeed, it won't be easy I guess. But sometimes the process you
>>> execute may
>>> > take quite some time and blocking the VM for that period is not a good
>>> > choice. So... what would you prefer to solve that? Do you prefer
>>> image-based
>>> > pooling as OSProcess does when IOPlugin is not used?
>>>
>>>
>>> How much might your work overlap with the Threaded FFI project for
>>> Cog?...
>>> ..."The Cog VM is single-threaded, providing green threads to the
>>> Smalltalk level, as do most Smalltalk VMs. There is a prototype
>>> subclass of the CoInterpreter, CoInterpreterMT, which provides a
>>> Python-like multi-threaded VM where any number of threads can share
>>> the VM with only one running the VM at any one time.  This uses an
>>> extremely simple low-level scheme for releasing and re-acquiring
>>> ownership of the VM designed by David Simmons.  This results in any
>>> and all FFI calls being non-blocking since there is such a low
>>> overhead to making a non-blocking FFI call that all FFI calls can be
>>> non-blocking.  Eliot has worked on this, implementing the prototype
>>> and the reentrant FFI plugin it requires.  But this needs to be
>>> 

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[p...@highoctane.be]

On Wed, Dec 23, 2015 at 4:08 AM, Mariano Martinez Peck <
marianop...@gmail.com> wrote:

>
>> >
>> >
>> > No, wait. You are getting down to the details so I want to discuss :)
>> >
>> > OSProcess does not use popen()/system() but rather fork()+exec(). That
>> means
>> > that OSProcess has implementing the very very low building blocks that
>> would
>> > allow you do to almost everything. But then it has it's complicated
>> matters.
>> > fork() + exec() cannot be implemented via FFI (this was already
>> discussed in
>> > another thread).
>>
>> Can you link that thread? I'm curious to learn this.
>
>
>
> It's a bit long discussion, but here it is:
> http://forum.world.st/ANN-Limbo-a-simple-shell-wrapper-for-NativeBoost-td4655894i20.html
>
>
>   Is it
>> multi-thread related?   I read [a]...
>> ..."The biggest problem with using fork+exec in this way is that you
>> can't guarantee *nothing* happens between the fork call and the exec
>> call. If, for example, the JVM decides to GC or move memory around,
>> you can have a fatal crash at the JVM process level. Because of that,
>> I don't recommend using fork + exec via FFI in JRuby, even though it's
>> pretty cool. "
>>
>>
>
> Yes, exactly. Pretty much the same. Since fork() clones and both parent
> and child execute same code (smalltalk vm in our case) until you call
> exec(), for that period of time, you will have the child running too, and
> that can screw up everything.
>
>
> ..."However, since this post I've learned of the "posix_spawn" [b]
>> function available on most Unix variants. It's basically fork + exec
>> in a single function, plus most of the typical security and IO tweaks
>> you might do after forking and before execing. It's definitely my
>> recommended alternative to fork+exec for JRuby."
>>
>
>
> *WOW. This seems to be one of the most useful recommendations I got. Thank
> you VERY MUCH.*
> I was thinking myself "how could I be the only one trying to call a
> fork+exec kind of function from FFI". Here is the answer.
> BTW...I have already started coding a FFI wrapper for such a function. I
> have very basic first command working :)
>
>
>>
>> And posix_spawn also seems recommended for use with the JVM [d]
>>
>
>
> Nice.
>
>
>> and that Windows and POSIX might have similar usage, and even better
>> match Windows CreateProcess than fork/exec [f] [g] [h]
>>
>> And since with 64-bit Spur we can expect very large images, spawn may
>> be preferred over fork/exec to minimize memory usage for creating
>> application subprocesses [i]
>>
>>
> Interesting.
>
>
>>
>> Fork versus threads section of [c]
>>
>> or should we be making use of pthread_atfork [e] ?
>>
>> Overall I learnt something new about forking in a multi-threaded
>> environment
>> "When fork() is called, only the calling thread is duplicated in the
>> child process."
>>
>
> I learned that today too :)
>
>
>>
>> [a]
>> http://blog.headius.com/2009/05/fork-and-exec-on-jvm-jruby-to-rescue.html
>> [b]
>> http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html
>> [d] http://skife.org/java/2012/01/24/java_daemonization_with_gressil.html
>> [c] http://repository.readscheme.org/ftp/papers/sw2002/gasbichler.pdf
>> [e]
>> http://stackoverflow.com/questions/4223553/c-does-exec-have-to-immediately-follow-fork-in-a-multi-threaded-process
>> [f]
>> http://stackoverflow.com/questions/33249253/what-is-the-difference-between-spawn-and-fork
>> [g] http://stackoverflow.com/questions/5883462/linux-createprocess
>> [h]
>> http://www.linuxquestions.org/questions/programming-9/fork-exec-versus-posix_spawn-835385/
>> [i]
>> http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html
>>
>>
> Thanks for all the links, very useful!
>
>
>> > However, there is a lot of other stuff that we believe it
>> > could. So... what we wanted to see if it is possible is to have one
>> only (or
>> > very few) primitives (like forkAndExec) and all the rest via FFI. And
>> then
>> > all the features I listed in the first email of this thread.
>> >
>> > With that in mind, we believe that:
>> >
>> > 1) Before starting dealing with forkAndExec, we could try a first
>> prototype
>> > using popen(). That would allow us test a 100% FFI based approeach, deal
>> > with writing and reading from pipes via stream, face the blocking
>> issue, see
>> > a prototype on how to read async from pipes etc etc etc. If we see this
>> > works nicely, on a second step we can build another approach that
>> instead of
>> > popen() uses forkAndExec primitive.
>>
>> Is popen() being restricted to unidirectional part of this consideration
>> [j] ?
>>
>>
>
> Yep. You can either define the read more or the write mode (stdout vs
> stdin) but not both, since popen() opens ONE pipe.
>
>
>
>> Also I read [j] that a problem with popen is that you do not get pid
>> of child so you cannot wait on command to complete.  Could posix_spawn
>> be preferrable to popen?
>>
>>
> Exactly. There is no standard way to get child pid from 

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Sven Van Caekenberghe]

> On 22 Dec 2015, at 19:30, Esteban Lorenzano  wrote:
> 
>> 
>> On 21 Dec 2015, at 20:55, Robert Withers  wrote:
>> 
>> 
>> 
>> On 12/21/2015 03:31 PM, Denis Kudriashov wrote:
>>> 
>>> 2015-12-21 19:07 GMT+01:00 Robert Withers :
>>> What are you and the RMOD team doing, I am curious?
>>> 
>>> RMOD is research team at INRIA university. We develop Pharo.
>> 
>> Yes, absolutely! I did not know it but I am glad to hear it. RMOD is the 
>> Pharo team.  You all keep going for excellence, every day.
> 
> No it is not :)
> the "Pharo team” includes many people outside RMoD:
> 
> - Sven
> - Doru and the glamour team (Andrei, Alexei and others I do not remember 
> their names, sorry… they work undercover)
> - Alain
> - People doing things in Lugano (like Yuri, who now is going to Bern)… 
> Tommasso, etc.
> - Or course Eliot who works on the VM (not just for Pharo).
> - And we also consider as part of the “pharo team”: Nicolai (Hess), Sean and 
> people who contributes a lot: Thierry, Norbert, Torsten (even if he probably 
> does not know we consider him part of the team :P), etc., etc., etc.
> - Essentially, if you contribute in any way with Pharo and you participate on 
> discussions, etc., you can consider yourself part of the Pharo team :)
> 
> Of course in the RMOD team there are people doing and contributing to Pharo 
> (but they do other things too, like Moose and even things nobody ever sees)… 
> Pharo is a much more complex and human-sparse project :)

Well said: Pharo is the result of a continuous effort of a large group of 
people. INRIA/RMOD is like the backbone.

It is dangerous to list names, because you always forget people. Like you did, 
but I am not going to try to correct you, because I will forget many people as 
well.

Every contribution: asking questions, entering in conversations, answering 
questions, helping others, writing blog posts and articles, reporting bugs, 
writing documentation, fixing bugs, it all counts, it all helps, a lot. 

> cheers!
> Esteban
> 
>> 
>> robert
>> -- 
>> . ..  ...   ^,^robert
>> Go Panthers!
>> 

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Robert Withers]

On 12/22/2015 03:12 PM, Esteban Lorenzano wrote:


On 22 Dec 2015, at 19:45, Robert Withers > wrote:


Hi Esteban,

I guess it sounded to me RMoD is the backbone team.  :)


I think you are into the VM, right?  Is there any chance you could 
build the pharo5.0 crypto plugins for me on 32bit Ubuntu? l I am 
unable to build myself right now. The package CryptographyPlugins on 
the Cryptography squeak source has them. This would really help me.


I am the overseer (like that guy in Fallout Shelter) ;)


Not sure but I can relate!

No, really: I’m a bit over everything and step into things when they 
need more time than the one people can usually invest in their free time.

Now is VM :)


And you are super busy with Spr, which I am looking forward to getting 
on my machine!



About your plugins: they are in my TODO list.
They will be eventually, but I cannot tell you exactly when 
(hopefully, soon (™)) :)


I liek eventually and have a great holidays. We've plenty of time and it 
is all temporary.  On reflection, what I would really and truly love is 
for Crypto and SecureSession, image code and plugins and tests all 
become a part of your CI. That is what I would really want for 
Christmas!  ;)



Thanks to everyone who is a part of being great!

best,
robert



cheers!
Esteban



thank you,
Robert

On 12/22/2015 01:30 PM, Esteban Lorenzano wrote:


On 21 Dec 2015, at 20:55, Robert Withers 
 wrote:




On 12/21/2015 03:31 PM, Denis Kudriashov wrote:


2015-12-21 19:07 GMT+01:00 Robert Withers 
>:


What are you and the RMOD team doing, I am curious?


RMOD is research team at INRIA university. We develop Pharo.


Yes, absolutely! I did not know it but I am glad to hear it. RMOD 
is the Pharo team.  You all keep going for excellence, every day.


No it is not :)
the "Pharo team” includes many people outside RMoD:

- Sven
- Doru and the glamour team (Andrei, Alexei and others I do not 
remember their names, sorry… they work undercover)

- Alain
- People doing things in Lugano (like Yuri, who now is going to 
Bern)… Tommasso, etc.

- Or course Eliot who works on the VM (not just for Pharo).
- And we also consider as part of the “pharo team”: Nicolai (Hess), 
Sean and people who contributes a lot: Thierry, Norbert, Torsten 
(even if he probably does not know we consider him part of the team 
:P), etc., etc., etc.
- Essentially, if you contribute in any way with Pharo and you 
participate on discussions, etc., you can consider yourself part of 
the Pharo team :)


Of course in the RMOD team there are people doing and contributing 
to Pharo (but they do other things too, like Moose and even things 
nobody ever sees)… Pharo is a much more complex and human-sparse 
project :)


cheers!
Esteban



robert
--
. ..  ...   ^,^robert
Go Panthers!




--
. ..  ...   ^,^robert
Go Panthers!




--
. ..  ...   ^,^robert
Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Esteban Lorenzano]

> On 21 Dec 2015, at 20:55, Robert Withers  wrote:
> 
> 
> 
> On 12/21/2015 03:31 PM, Denis Kudriashov wrote:
>> 
>> 2015-12-21 19:07 GMT+01:00 Robert Withers > >:
>> What are you and the RMOD team doing, I am curious?
>> 
>> RMOD is research team at INRIA university. We develop Pharo.
> 
> Yes, absolutely! I did not know it but I am glad to hear it. RMOD is the 
> Pharo team.  You all keep going for excellence, every day.

No it is not :)
the "Pharo team” includes many people outside RMoD:

- Sven
- Doru and the glamour team (Andrei, Alexei and others I do not remember their 
names, sorry… they work undercover)
- Alain
- People doing things in Lugano (like Yuri, who now is going to Bern)… 
Tommasso, etc.
- Or course Eliot who works on the VM (not just for Pharo).
- And we also consider as part of the “pharo team”: Nicolai (Hess), Sean and 
people who contributes a lot: Thierry, Norbert, Torsten (even if he probably 
does not know we consider him part of the team :P), etc., etc., etc.
- Essentially, if you contribute in any way with Pharo and you participate on 
discussions, etc., you can consider yourself part of the Pharo team :)

Of course in the RMOD team there are people doing and contributing to Pharo 
(but they do other things too, like Moose and even things nobody ever sees)… 
Pharo is a much more complex and human-sparse project :)

cheers!
Esteban

> 
> robert
> -- 
> . ..  ...   ^,^robert
> Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Robert Withers]

Hi Esteban,

I guess it sounded to me RMoD is the backbone team.  :)


I think you are into the VM, right?  Is there any chance you could build 
the pharo5.0 crypto plugins for me on 32bit Ubuntu? l I am unable to 
build myself right now. The package CryptographyPlugins on the 
Cryptography squeak source has them. This would really help me.


thank you,
Robert

On 12/22/2015 01:30 PM, Esteban Lorenzano wrote:


On 21 Dec 2015, at 20:55, Robert Withers > wrote:




On 12/21/2015 03:31 PM, Denis Kudriashov wrote:


2015-12-21 19:07 GMT+01:00 Robert Withers 
>:


What are you and the RMOD team doing, I am curious?


RMOD is research team at INRIA university. We develop Pharo.


Yes, absolutely! I did not know it but I am glad to hear it. RMOD is 
the Pharo team.  You all keep going for excellence, every day.


No it is not :)
the "Pharo team” includes many people outside RMoD:

- Sven
- Doru and the glamour team (Andrei, Alexei and others I do not 
remember their names, sorry… they work undercover)

- Alain
- People doing things in Lugano (like Yuri, who now is going to Bern)… 
Tommasso, etc.

- Or course Eliot who works on the VM (not just for Pharo).
- And we also consider as part of the “pharo team”: Nicolai (Hess), 
Sean and people who contributes a lot: Thierry, Norbert, Torsten (even 
if he probably does not know we consider him part of the team :P), 
etc., etc., etc.
- Essentially, if you contribute in any way with Pharo and you 
participate on discussions, etc., you can consider yourself part of 
the Pharo team :)


Of course in the RMOD team there are people doing and contributing to 
Pharo (but they do other things too, like Moose and even things nobody 
ever sees)… Pharo is a much more complex and human-sparse project :)


cheers!
Esteban



robert
--
. ..  ...   ^,^robert
Go Panthers!




--
. ..  ...   ^,^robert
Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Esteban Lorenzano]

> On 22 Dec 2015, at 19:45, Robert Withers  wrote:
> 
> Hi Esteban,
> 
> I guess it sounded to me RMoD is the backbone team.  :)
> 
> 
> I think you are into the VM, right?  Is there any chance you could build the 
> pharo5.0 crypto plugins for me on 32bit Ubuntu? l I am unable to build myself 
> right now. The package CryptographyPlugins on the Cryptography squeak source 
> has them. This would really help me.

I am the overseer (like that guy in Fallout Shelter) ;)
No, really: I’m a bit over everything and step into things when they need more 
time than the one people can usually invest in their free time. 
Now is VM :)
About your plugins: they are in my TODO list. 
They will be eventually, but I cannot tell you exactly when (hopefully, soon 
(™)) :)

cheers!
Esteban

> 
> thank you,
> Robert
> 
> On 12/22/2015 01:30 PM, Esteban Lorenzano wrote:
>> 
>>> On 21 Dec 2015, at 20:55, Robert Withers < 
>>> robert.w.with...@gmail.com 
>>> > wrote:
>>> 
>>> 
>>> 
>>> On 12/21/2015 03:31 PM, Denis Kudriashov wrote:
 
 2015-12-21 19:07 GMT+01:00 Robert Withers >:
 What are you and the RMOD team doing, I am curious?
 
 RMOD is research team at INRIA university. We develop Pharo.
>>> 
>>> Yes, absolutely! I did not know it but I am glad to hear it. RMOD is the 
>>> Pharo team.  You all keep going for excellence, every day.
>> 
>> No it is not :)
>> the "Pharo team” includes many people outside RMoD:
>> 
>> - Sven
>> - Doru and the glamour team (Andrei, Alexei and others I do not remember 
>> their names, sorry… they work undercover)
>> - Alain
>> - People doing things in Lugano (like Yuri, who now is going to Bern)… 
>> Tommasso, etc.
>> - Or course Eliot who works on the VM (not just for Pharo).
>> - And we also consider as part of the “pharo team”: Nicolai (Hess), Sean and 
>> people who contributes a lot: Thierry, Norbert, Torsten (even if he probably 
>> does not know we consider him part of the team :P), etc., etc., etc.
>> - Essentially, if you contribute in any way with Pharo and you participate 
>> on discussions, etc., you can consider yourself part of the Pharo team :)
>> 
>> Of course in the RMOD team there are people doing and contributing to Pharo 
>> (but they do other things too, like Moose and even things nobody ever sees)… 
>> Pharo is a much more complex and human-sparse project :)
>> 
>> cheers!
>> Esteban
>> 
>>> 
>>> robert
>>> -- 
>>> . ..  ...   ^,^robert
>>> Go Panthers!
>> 
> 
> -- 
> . ..  ...   ^,^robert
> Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Esteban Lorenzano]

yeah, I’m aware of that but I wanted to express what we feel about being “pharo 
team” and I thought naming people (even if, of course, is far from exhaustive) 
was more explicit. 

cheers,
Esteban

> On 22 Dec 2015, at 19:28, Sven Van Caekenberghe  wrote:
> 
>> 
>> On 22 Dec 2015, at 19:30, Esteban Lorenzano  wrote:
>> 
>>> 
>>> On 21 Dec 2015, at 20:55, Robert Withers  wrote:
>>> 
>>> 
>>> 
>>> On 12/21/2015 03:31 PM, Denis Kudriashov wrote:
 
 2015-12-21 19:07 GMT+01:00 Robert Withers :
 What are you and the RMOD team doing, I am curious?
 
 RMOD is research team at INRIA university. We develop Pharo.
>>> 
>>> Yes, absolutely! I did not know it but I am glad to hear it. RMOD is the 
>>> Pharo team.  You all keep going for excellence, every day.
>> 
>> No it is not :)
>> the "Pharo team” includes many people outside RMoD:
>> 
>> - Sven
>> - Doru and the glamour team (Andrei, Alexei and others I do not remember 
>> their names, sorry… they work undercover)
>> - Alain
>> - People doing things in Lugano (like Yuri, who now is going to Bern)… 
>> Tommasso, etc.
>> - Or course Eliot who works on the VM (not just for Pharo).
>> - And we also consider as part of the “pharo team”: Nicolai (Hess), Sean and 
>> people who contributes a lot: Thierry, Norbert, Torsten (even if he probably 
>> does not know we consider him part of the team :P), etc., etc., etc.
>> - Essentially, if you contribute in any way with Pharo and you participate 
>> on discussions, etc., you can consider yourself part of the Pharo team :)
>> 
>> Of course in the RMOD team there are people doing and contributing to Pharo 
>> (but they do other things too, like Moose and even things nobody ever sees)… 
>> Pharo is a much more complex and human-sparse project :)
> 
> Well said: Pharo is the result of a continuous effort of a large group of 
> people. INRIA/RMOD is like the backbone.
> 
> It is dangerous to list names, because you always forget people. Like you 
> did, but I am not going to try to correct you, because I will forget many 
> people as well.
> 
> Every contribution: asking questions, entering in conversations, answering 
> questions, helping others, writing blog posts and articles, reporting bugs, 
> writing documentation, fixing bugs, it all counts, it all helps, a lot. 
> 
>> cheers!
>> Esteban
>> 
>>> 
>>> robert
>>> -- 
>>> . ..  ...   ^,^robert
>>> Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Mariano Martinez Peck]

>
>
> >
> >
> > No, wait. You are getting down to the details so I want to discuss :)
> >
> > OSProcess does not use popen()/system() but rather fork()+exec(). That
> means
> > that OSProcess has implementing the very very low building blocks that
> would
> > allow you do to almost everything. But then it has it's complicated
> matters.
> > fork() + exec() cannot be implemented via FFI (this was already
> discussed in
> > another thread).
>
> Can you link that thread? I'm curious to learn this.



It's a bit long discussion, but here it is:
http://forum.world.st/ANN-Limbo-a-simple-shell-wrapper-for-NativeBoost-td4655894i20.html


  Is it
> multi-thread related?   I read [a]...
> ..."The biggest problem with using fork+exec in this way is that you
> can't guarantee *nothing* happens between the fork call and the exec
> call. If, for example, the JVM decides to GC or move memory around,
> you can have a fatal crash at the JVM process level. Because of that,
> I don't recommend using fork + exec via FFI in JRuby, even though it's
> pretty cool. "
>
>

Yes, exactly. Pretty much the same. Since fork() clones and both parent and
child execute same code (smalltalk vm in our case) until you call exec(),
for that period of time, you will have the child running too, and that can
screw up everything.


..."However, since this post I've learned of the "posix_spawn" [b]
> function available on most Unix variants. It's basically fork + exec
> in a single function, plus most of the typical security and IO tweaks
> you might do after forking and before execing. It's definitely my
> recommended alternative to fork+exec for JRuby."
>


*WOW. This seems to be one of the most useful recommendations I got. Thank
you VERY MUCH.*
I was thinking myself "how could I be the only one trying to call a
fork+exec kind of function from FFI". Here is the answer.
BTW...I have already started coding a FFI wrapper for such a function. I
have very basic first command working :)


>
> And posix_spawn also seems recommended for use with the JVM [d]
>


Nice.


> and that Windows and POSIX might have similar usage, and even better
> match Windows CreateProcess than fork/exec [f] [g] [h]
>
> And since with 64-bit Spur we can expect very large images, spawn may
> be preferred over fork/exec to minimize memory usage for creating
> application subprocesses [i]
>
>
Interesting.


>
> Fork versus threads section of [c]
>
> or should we be making use of pthread_atfork [e] ?
>
> Overall I learnt something new about forking in a multi-threaded
> environment
> "When fork() is called, only the calling thread is duplicated in the
> child process."
>

I learned that today too :)


>
> [a]
> http://blog.headius.com/2009/05/fork-and-exec-on-jvm-jruby-to-rescue.html
> [b]
> http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html
> [d] http://skife.org/java/2012/01/24/java_daemonization_with_gressil.html
> [c] http://repository.readscheme.org/ftp/papers/sw2002/gasbichler.pdf
> [e]
> http://stackoverflow.com/questions/4223553/c-does-exec-have-to-immediately-follow-fork-in-a-multi-threaded-process
> [f]
> http://stackoverflow.com/questions/33249253/what-is-the-difference-between-spawn-and-fork
> [g] http://stackoverflow.com/questions/5883462/linux-createprocess
> [h]
> http://www.linuxquestions.org/questions/programming-9/fork-exec-versus-posix_spawn-835385/
> [i]
> http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html
>
>
Thanks for all the links, very useful!


> > However, there is a lot of other stuff that we believe it
> > could. So... what we wanted to see if it is possible is to have one only
> (or
> > very few) primitives (like forkAndExec) and all the rest via FFI. And
> then
> > all the features I listed in the first email of this thread.
> >
> > With that in mind, we believe that:
> >
> > 1) Before starting dealing with forkAndExec, we could try a first
> prototype
> > using popen(). That would allow us test a 100% FFI based approeach, deal
> > with writing and reading from pipes via stream, face the blocking issue,
> see
> > a prototype on how to read async from pipes etc etc etc. If we see this
> > works nicely, on a second step we can build another approach that
> instead of
> > popen() uses forkAndExec primitive.
>
> Is popen() being restricted to unidirectional part of this consideration
> [j] ?
>
>

Yep. You can either define the read more or the write mode (stdout vs
stdin) but not both, since popen() opens ONE pipe.



> Also I read [j] that a problem with popen is that you do not get pid
> of child so you cannot wait on command to complete.  Could posix_spawn
> be preferrable to popen?
>
>
Exactly. There is no standard way to get child pid from popen() so you
can't do polling in image side to wait it's completion.
So yeah, posix_spawn() does fix this issue as well as many many other.


> and just btw, mildly interesting is using sockets rather than pipes
> for interprocess communication [l]
>
> 

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Denis Kudriashov]

2015-12-21 19:07 GMT+01:00 Robert Withers :

> What are you and the RMOD team doing, I am curious?


RMOD is research team at INRIA university. We develop Pharo.

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Robert Withers]

On 12/21/2015 01:34 PM, stepharo wrote:



Le 19/12/15 14:33, Dimitris Chloupis a écrit :

I am very excited for such a project.

I hope you manage to fix the stdout problem on macos.

One thing I would like to see is an expansion of the ability of 
OSProcess to launch multiple instances of pharo and make them 
communicate with each other. Thats a way to have pharo running on 
multiple cores. Maybe establish a framework of communication of 
images that will be based on the existing design of the fork 
mechanism in Pharo.


Python is doing something similar with multiprocessing module  which 
launches multiple processes of cpython executable and makes them 
communication with each other through a similar api to its threading 
module. This way python takes advantage of multiple cores. Python 
threads like pharo are green threads and pharo like python has a GIL 
like mechanism.


this will be probably another project.


Well, if you take Seamless or SecureSession and conenct 4 VMs together 
and stretch the heap safely, what's the difference? You can crash one of 
them and NOT affect the others, so it's all good.



robert



Stef


Taking into account that current computers have at least 4 cores, its 
not a bad idea to speed up pharo like this at least  400% ;)


On Sat, Dec 19, 2015 at 3:24 PM Mariano Martinez Peck 
> wrote:


Dear all,

I am tremendously happy to announce you that Pharo Consortium
will sponsor yet another development effort. In this particular
case, it's my honor to carry on such an effort and I will be
developing and improving a few things and ideas. All the contract
and paperwork has already been done so it's time to start working.

Regarding the developments itself, we discussed about these topics:

1) Experiment with a simpler yet more limited OSProcess
alternative to execute OS commands.
2) Improving FileSystem in order to better deal with some POSIX
stuff like symbolic links, unix file permissions, etc etc.

I have already started with 1). OSProcess is super complete and
it involves some packages (OSProcess / CommandShell) as well as
some VM plugins (OSProcessPlugin, AIOPlugin). OSProcess provides
lots of features (like forking the running image) but we would
like to focus only in executing OS commands. In addition,
OSProcess dates from quite some years ago when many of the
current infrastructure features did not exist yet.

So the idea is to think a simpler alternative to execute OS
commands, using new features such as threaded FFI (for example,
for reading async from pipes), pinned objects, etc. We want to
use FFI as much as possible rather than VM plugins.

From the image side, we are thinking about an API which would
look like a builder-like API (FileSystem, XStream, etc).

We have been discussing with David Lewis (OSProcess author) as
well as with Eliot Miranda, Esteban Lorenzano, Damien Pollet,
Stephane Ducasse, etc in order to agree in a project that would
be worth doing (otherwise we would continue using existing
tools). And they have all been very kind with me providing
positive discussions.

I will soon do a survey to measure the most common use cases of
OSProcess and related tools.

If you have any thought, question, feedback, or whatever, please
let us know.

Finally, note that I will be doing this project together with
another client's project (Quuve) and my life does not have much
free time these days...so please be patient!!!

Best,

-- 
Mariano

http://marianopeck.wordpress.com





--
. ..  ...   ^,^robert
Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Robert Withers]

On 12/21/2015 03:31 PM, Denis Kudriashov wrote:


2015-12-21 19:07 GMT+01:00 Robert Withers >:


What are you and the RMOD team doing, I am curious?


RMOD is research team at INRIA university. We develop Pharo.


Yes, absolutely! I did not know it but I am glad to hear it. RMOD is the 
Pharo team.  You all keep going for excellence, every day.


robert

--
. ..  ...   ^,^robert
Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Robert Withers]

Well, your countries have been intricately connected, and once again 
north of Sykes-Picot. Kak dela? Karosho.  I'm more Germany/Egypt connected.


What are you and the RMOD team doing, I am curious?

Ok, I don't mean to drag us to politics, nor religion. I was making a 
point that connects Pharo to 21st century computing which will challenge 
governments, so it is related.  Yet, I'll keep it on slack 
#randomrandom! ;-)


cheers,
robert

On 12/21/2015 12:49 PM, Denis Kudriashov wrote:

I'm from Russia. Now I work in Rmod team and live in France.
But let's talk here about Pharo.

2015-12-21 18:03 GMT+01:00 Robert Withers >:


Out of curiosity and to make a point, where are you from, Denis?
Where do you live now? Many from different lands, closer to each
other than eah to our own governments. Trending into the future,
transnational connections will deviate from their (governments') norm.

On 12/21/2015 10:17 AM, Denis Kudriashov wrote:


2015-12-21 11:19 GMT+01:00 Robert Withers
>:

governments cannot crack it.


Why it is disadvantage?


-- 
. ..  ...   ^,^robert

Go Panthers!




--
. ..  ...   ^,^robert
Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Robert Withers]

On 12/21/2015 12:55 PM, Sven Van Caekenberghe wrote:

On 21 Dec 2015, at 17:20, Robert Withers  wrote:



On 12/21/2015 10:17 AM, Denis Kudriashov wrote:

2015-12-21 11:19 GMT+01:00 Robert Withers :
governments cannot crack it.

Why it is disadvantage?

Governments stay in power, or exercise collective sovereignty, through 
dominion, agency and possession. Loss of control over information, and by 
extension currency, affects possession, or having control. They still have 
dominion and agency, but without three legs you can build no stool to sit upon. 
If loss of control over information and currency undermines possession, then 
the concept of the modern nation-state is undermined. I view this as a negative.

This is not the place for politics, but cryptography is inherently linked to it.


Yes, exactly.


I must say that it really scares me if someone implementing all kinds of 
cryptography thinks that 'the government' should be able to crack it. If that 
is your opinion (I might understand you wrongly), than just write everything in 
plain text and be done with it.


Firstly, we learn: DON'T PANIC!  ;)  Seriously, don't be scared.  I did 
not say I thought government should crack it, just that I can view it as 
a negative. They should absolutely NOT be able to crack it, but we'd 
better come up with a plan.






--
. ..  ...   ^,^robert
Go Panthers!





--
. ..  ...   ^,^robert
Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Thierry Goubier]

Le 21/12/2015 11:05, Ben Coman a écrit :

On Mon, Dec 21, 2015 at 8:30 PM, Denis Kudriashov  wrote:

Hello

2015-12-20 19:54 GMT+01:00 Mariano Martinez Peck :


That being said, I of course like the idea of distributing processing
across multiple images. No wonder, my PhD tool (Marea) and related
sub-projects (Fuel, Ghost proxies, etc) have something to do with the
mentioned topic. Dave already mentioned about RemoteTask, which was
something I was about to mention. There is also Seamless project (now used
by Remote Debugger stuff) and quite other tools. Finally, I think this
project may have a big percentage that could be considered as "research". So
having a PhD / master student working on this would also be nice. In fact,
maybe Denis Kudriashov might work on that topic??? (he is now in RMOD team).



Yes. This is what I'm doing now. At last weeks of this year I plan to finish
my improvements to Seamless. Then I will adapt system and tools for remote
usage. Main goal is allow work with remote images same way we work locally:
remote debugging, browsing, any kind of system navigation, editing code.


Cool.  I'm looking forward to maybe one day working like that with a
minimal image on embedded hardware.


With ARM and MIPS support coming, this looks closer than ever :)

Thierry

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Robert Withers]

On 12/21/2015 05:05 AM, Ben Coman wrote:

On Mon, Dec 21, 2015 at 8:30 PM, Denis Kudriashov  wrote:

Hello

2015-12-20 19:54 GMT+01:00 Mariano Martinez Peck :

That being said, I of course like the idea of distributing processing
across multiple images. No wonder, my PhD tool (Marea) and related
sub-projects (Fuel, Ghost proxies, etc) have something to do with the
mentioned topic. Dave already mentioned about RemoteTask, which was
something I was about to mention. There is also Seamless project (now used
by Remote Debugger stuff) and quite other tools. Finally, I think this
project may have a big percentage that could be considered as "research". So
having a PhD / master student working on this would also be nice. In fact,
maybe Denis Kudriashov might work on that topic??? (he is now in RMOD team).


Yes. This is what I'm doing now. At last weeks of this year I plan to finish
my improvements to Seamless. Then I will adapt system and tools for remote
usage. Main goal is allow work with remote images same way we work locally:
remote debugging, browsing, any kind of system navigation, editing code.

Cool.  I'm looking forward to maybe one day working like that with a
minimal image on embedded hardware.


I look forward to it. There's some set of capabilities you can't get 
with your daily java or any non-image based dev environment, which 
leaves Pharo ruling the cloud. Give it a few years.


Have you thought about SecureSession as the session solution? It does 
have advantages and disadvantages, if I may briefly proselytize, 
advertise and pitch you on this good news of yours:


advantages:
a) no certificates used
b) no third parties involved
c) highly secure
d) performant with crypto plugins
e) code visibility: entirely in squeak, no platform code requirements
f) community support of native code
g) forward error correction and compression on the horizon, hopefully.

disadvantages:
a) not performant without plugins
b) in further development
c) protocol likely will change in a coordinated fashion
d) non-standard protocol
e) non-validated by appropriate federal validators, as good crypto
f) governments cannot crack it.
g) may be slower than existing solution.

there 7 for and 7 against for a neutral honest appraisal from the 
author. Fair dinkum.


--
. ..  ...   ^,^robert
Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[p...@highoctane.be]

Love to read that!

On Mon, Dec 21, 2015 at 10:30 AM, Denis Kudriashov 
wrote:

> Hello
>
> 2015-12-20 19:54 GMT+01:00 Mariano Martinez Peck :
>
>> That being said, I of course like the idea of distributing processing
>> across multiple images. No wonder, my PhD tool (Marea) and related
>> sub-projects (Fuel, Ghost proxies, etc) have something to do with the
>> mentioned topic. Dave already mentioned about RemoteTask, which was
>> something I was about to mention. There is also Seamless project (now used
>> by Remote Debugger stuff) and quite other tools. Finally, I think this
>> project may have a big percentage that could be considered as "research".
>> So having a PhD / master student working on this would also be nice. In
>> fact, maybe Denis Kudriashov might work on that topic??? (he is now in
>> RMOD team).
>
>
> Yes. This is what I'm doing now. At last weeks of this year I plan to
> finish my improvements to Seamless. Then I will adapt system and tools for
> remote usage. Main goal is allow work with remote images same way we work
> locally: remote debugging, browsing, any kind of system navigation, editing
> code.
>

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Denis Kudriashov]

Hello

2015-12-20 19:54 GMT+01:00 Mariano Martinez Peck :

> That being said, I of course like the idea of distributing processing
> across multiple images. No wonder, my PhD tool (Marea) and related
> sub-projects (Fuel, Ghost proxies, etc) have something to do with the
> mentioned topic. Dave already mentioned about RemoteTask, which was
> something I was about to mention. There is also Seamless project (now used
> by Remote Debugger stuff) and quite other tools. Finally, I think this
> project may have a big percentage that could be considered as "research".
> So having a PhD / master student working on this would also be nice. In
> fact, maybe Denis Kudriashov might work on that topic??? (he is now in
> RMOD team).


Yes. This is what I'm doing now. At last weeks of this year I plan to
finish my improvements to Seamless. Then I will adapt system and tools for
remote usage. Main goal is allow work with remote images same way we work
locally: remote debugging, browsing, any kind of system navigation, editing
code.

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Ben Coman]

On Mon, Dec 21, 2015 at 8:30 PM, Denis Kudriashov  wrote:
> Hello
>
> 2015-12-20 19:54 GMT+01:00 Mariano Martinez Peck :
>>
>> That being said, I of course like the idea of distributing processing
>> across multiple images. No wonder, my PhD tool (Marea) and related
>> sub-projects (Fuel, Ghost proxies, etc) have something to do with the
>> mentioned topic. Dave already mentioned about RemoteTask, which was
>> something I was about to mention. There is also Seamless project (now used
>> by Remote Debugger stuff) and quite other tools. Finally, I think this
>> project may have a big percentage that could be considered as "research". So
>> having a PhD / master student working on this would also be nice. In fact,
>> maybe Denis Kudriashov might work on that topic??? (he is now in RMOD team).
>
>
> Yes. This is what I'm doing now. At last weeks of this year I plan to finish
> my improvements to Seamless. Then I will adapt system and tools for remote
> usage. Main goal is allow work with remote images same way we work locally:
> remote debugging, browsing, any kind of system navigation, editing code.

Cool.  I'm looking forward to maybe one day working like that with a
minimal image on embedded hardware.

cheers -ben

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Ben Coman]

On Tue, Dec 22, 2015 at 3:20 AM, Mariano Martinez Peck
 wrote:
>
>
> On Sun, Dec 20, 2015 at 5:21 PM, Thierry Goubier 
> wrote:
>>
>> Hi Mariano,
>>
>> Le 20/12/2015 19:54, Mariano Martinez Peck a écrit :
>>>
>>> Hi Dimitri, Dave et all.
>>>
>>> First let me do a little disclaimer. Regarding the developments itself
>>> of this sponsorship, we discussed about the topics I said in the first
>>> email. However, those are not written in stone and that's why I also
>>> asked for feedback. As you know, Pharo has relatively little resources
>>> so those should be well spent. If we decide to take a spin into a
>>> project X, it doesn't mean other projects are not worth. It simply means
>>> that we think X will be the most useful for the time being.
>>>
>>> One of the final goals of the discusses topics, is to improve a bit the
>>> infrastructure for using Pharo as scripting (imagine Coral). For such a
>>> thing, we believe that improving FileSystem and OSProcess-like is a good
>>> first step to improve the underlying infrastructure.
>>
>>
>> Would we have access then to a command-line oriented minimal Pharo image?
>>
>
> Well, just for putting perspective my sponsorship is about 3 full months of
> work effort. It doesn't mean I won't work beyond that. But to give you an
> idea. I think we all want minimal images , command line etc. And Pharo has
> already done a huge effort in that direction.

>>>
>>> That being said, I of course like the idea of distributing processing
>>> across multiple images. No wonder, my PhD tool (Marea) and related
>>> sub-projects (Fuel, Ghost proxies, etc) have something to do with the
>>> mentioned topic. Dave already mentioned about RemoteTask, which was
>>> something I was about to mention. There is also Seamless project (now
>>> used by Remote Debugger stuff) and quite other tools. Finally, I think
>>> this project may have a big percentage that could be considered as
>>> "research". So having a PhD / master student working on this would also
>>> be nice. In fact, maybe Denis Kudriashov might work on that topic??? (he
>>> is now in RMOD team).
>>>
>>> Also, not that forking the image is not my main short-term goal. As
>>> said, I will experiment first with a simple tool to execute OS commands
>>> that would be FFI based, trying to make async operations, and that would
>>> help in most cases. Maybe for those that are not supported, then
>>> fallback in OSProcess. Note that Dave (OSProcess author) is very helpful
>>> with me and we are discussing together about this.
>>
>>
>> My guess is that, at least on Linux/Unix, that FFI will be exactly the
>> same as the one used/expressed by the OSProcess plugin, that is
>> popen/system.
>
>
> No, wait. You are getting down to the details so I want to discuss :)
>
> OSProcess does not use popen()/system() but rather fork()+exec(). That means
> that OSProcess has implementing the very very low building blocks that would
> allow you do to almost everything. But then it has it's complicated matters.
> fork() + exec() cannot be implemented via FFI (this was already discussed in
> another thread).

Can you link that thread? I'm curious to learn this.  Is it
multi-thread related?   I read [a]...
..."The biggest problem with using fork+exec in this way is that you
can't guarantee *nothing* happens between the fork call and the exec
call. If, for example, the JVM decides to GC or move memory around,
you can have a fatal crash at the JVM process level. Because of that,
I don't recommend using fork + exec via FFI in JRuby, even though it's
pretty cool. "

..."However, since this post I've learned of the "posix_spawn" [b]
function available on most Unix variants. It's basically fork + exec
in a single function, plus most of the typical security and IO tweaks
you might do after forking and before execing. It's definitely my
recommended alternative to fork+exec for JRuby."

And posix_spawn also seems recommended for use with the JVM [d]
and that Windows and POSIX might have similar usage, and even better
match Windows CreateProcess than fork/exec [f] [g] [h]

And since with 64-bit Spur we can expect very large images, spawn may
be preferred over fork/exec to minimize memory usage for creating
application subprocesses [i]


Fork versus threads section of [c]

or should we be making use of pthread_atfork [e] ?

Overall I learnt something new about forking in a multi-threaded environment
"When fork() is called, only the calling thread is duplicated in the
child process."

[a] http://blog.headius.com/2009/05/fork-and-exec-on-jvm-jruby-to-rescue.html
[b] http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_spawn.html
[d] http://skife.org/java/2012/01/24/java_daemonization_with_gressil.html
[c] http://repository.readscheme.org/ftp/papers/sw2002/gasbichler.pdf
[e] 
http://stackoverflow.com/questions/4223553/c-does-exec-have-to-immediately-follow-fork-in-a-multi-threaded-process
[f] 

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Robert Withers]

On 12/21/2015 10:17 AM, Denis Kudriashov wrote:


2015-12-21 11:19 GMT+01:00 Robert Withers >:


governments cannot crack it.


Why it is disadvantage?


Governments stay in power, or exercise collective sovereignty, through 
dominion, agency and possession. Loss of control over information, and 
by extension currency, affects possession, or having control. They still 
have dominion and agency, but without three legs you can build no stool 
to sit upon. If loss of control over information and currency undermines 
possession, then the concept of the modern nation-state is undermined. I 
view this as a negative.


--
. ..  ...   ^,^robert
Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Mariano Martinez Peck]

On Sun, Dec 20, 2015 at 5:21 PM, Thierry Goubier 
wrote:

> Hi Mariano,
>
> Le 20/12/2015 19:54, Mariano Martinez Peck a écrit :
>
>> Hi Dimitri, Dave et all.
>>
>> First let me do a little disclaimer. Regarding the developments itself
>> of this sponsorship, we discussed about the topics I said in the first
>> email. However, those are not written in stone and that's why I also
>> asked for feedback. As you know, Pharo has relatively little resources
>> so those should be well spent. If we decide to take a spin into a
>> project X, it doesn't mean other projects are not worth. It simply means
>> that we think X will be the most useful for the time being.
>>
>> One of the final goals of the discusses topics, is to improve a bit the
>> infrastructure for using Pharo as scripting (imagine Coral). For such a
>> thing, we believe that improving FileSystem and OSProcess-like is a good
>> first step to improve the underlying infrastructure.
>>
>
> Would we have access then to a command-line oriented minimal Pharo image?
>
>
Well, just for putting perspective my sponsorship is about 3 full months of
work effort. It doesn't mean I won't work beyond that. But to give you an
idea. I think we all want minimal images , command line etc. And Pharo has
already done a huge effort in that direction.



> That being said, I of course like the idea of distributing processing
>> across multiple images. No wonder, my PhD tool (Marea) and related
>> sub-projects (Fuel, Ghost proxies, etc) have something to do with the
>> mentioned topic. Dave already mentioned about RemoteTask, which was
>> something I was about to mention. There is also Seamless project (now
>> used by Remote Debugger stuff) and quite other tools. Finally, I think
>> this project may have a big percentage that could be considered as
>> "research". So having a PhD / master student working on this would also
>> be nice. In fact, maybe Denis Kudriashov might work on that topic??? (he
>> is now in RMOD team).
>>
>> Also, not that forking the image is not my main short-term goal. As
>> said, I will experiment first with a simple tool to execute OS commands
>> that would be FFI based, trying to make async operations, and that would
>> help in most cases. Maybe for those that are not supported, then
>> fallback in OSProcess. Note that Dave (OSProcess author) is very helpful
>> with me and we are discussing together about this.
>>
>
> My guess is that, at least on Linux/Unix, that FFI will be exactly the
> same as the one used/expressed by the OSProcess plugin, that is
> popen/system.
>

No, wait. You are getting down to the details so I want to discuss :)

OSProcess does not use popen()/system() but rather fork()+exec(). That
means that OSProcess has implementing the very very low building blocks
that would allow you do to almost everything. But then it has it's
complicated matters. fork() + exec() cannot be implemented via FFI (this
was already discussed in another thread). However, there is a lot of other
stuff that we believe it could. So... what we wanted to see if it is
possible is to have one only (or very few) primitives (like forkAndExec)
and all the rest via FFI. And then all the features I listed in the first
email of this thread.

With that in mind, we believe that:

1) Before starting dealing with forkAndExec, we could try a first prototype
using popen(). That would allow us test a 100% FFI based approeach, deal
with writing and reading from pipes via stream, face the blocking issue,
see a prototype on how to read async from pipes etc etc etc. If we see this
works nicely, on a second step we can build another approach that instead
of popen() uses forkAndExec primitive.

2) We believe that there is a large number of OSProcess users that use
OSProcess only for executing OS commands AND popen() approach would be more
than enough (we will do a survey about this). So, a very simply solution
via FFI for these users rather than having all OSProcess, Command Shell,
the plugins etc, may be worth.

What do you think?


> Now, given some of the difficulties seen when dealing with external
> processes, I'd be wary of the "async" side of things. Random failures
> seemingly linked to signal loss are certainly not something I'd like to
> debug.
>
>
Indeed, it won't be easy I guess. But sometimes the process you execute may
take quite some time and blocking the VM for that period is not a good
choice. So... what would you prefer to solve that? Do you prefer
image-based pooling as OSProcess does when IOPlugin is not used?


> Something interesting and powerful would be pseudo-tty support. It would
> allow a Pharo image to maintain an open connection to a shell, for example,
> reducing the cost and latency of launching external commands and
> maintaining a context.
>

Interesting!

-- 
Mariano
http://marianopeck.wordpress.com

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Denis Kudriashov]

2015-12-21 11:19 GMT+01:00 Robert Withers :

> governments cannot crack it.


Why it is disadvantage?

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Denis Kudriashov]

I'm from Russia. Now I work in Rmod team and live in France.
But let's talk here about Pharo.

2015-12-21 18:03 GMT+01:00 Robert Withers :

> Out of curiosity and to make a point, where are you from, Denis? Where do
> you live now? Many from different lands, closer to each other than eah to
> our own governments. Trending into the future, transnational connections
> will deviate from their (governments') norm.
>
> On 12/21/2015 10:17 AM, Denis Kudriashov wrote:
>
>
> 2015-12-21 11:19 GMT+01:00 Robert Withers :
>
>> governments cannot crack it.
>
>
> Why it is disadvantage?
>
>
> --
> . ..  ...   ^,^robert
> Go Panthers!
>
>

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Robert Withers]

Out of curiosity and to make a point, where are you from, Denis? Where 
do you live now? Many from different lands, closer to each other than 
eah to our own governments. Trending into the future, transnational 
connections will deviate from their (governments') norm.


On 12/21/2015 10:17 AM, Denis Kudriashov wrote:


2015-12-21 11:19 GMT+01:00 Robert Withers >:


governments cannot crack it.


Why it is disadvantage?


--
. ..  ...   ^,^robert
Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Robert Withers]

Out of curiosity and to make a point, where are you from, Denis? Where 
do you live now? Many from different lands, closer to each other than 
eah to our own governments. Trending into the future, transnational 
connections will deviate from their (governments') norm.


On 12/21/2015 10:17 AM, Denis Kudriashov wrote:


2015-12-21 11:19 GMT+01:00 Robert Withers >:


governments cannot crack it.


Why it is disadvantage?


--
. ..  ...   ^,^robert
Go Panthers!

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Pierce Ng]

On Sun, Dec 20, 2015 at 08:56:43AM +, Dimitris Chloupis wrote:
> Fuel I assume enters here the equation as a data exchange format , the
> problem I have with fuel is that its not backward compatible which for me

Once you have a set of OS processes running Pharo how they talk to each other
is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC, JSONRPC,
msgpack, coordinate via Redis/SQLite/PostgreSQL, etc.

Pierce

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Dimitris Chloupis]

I mentions this because in OSProcess-Unix there is UnixProcess class, at
the class side there are several examples that launch a separate instance
of Squeak/Pharo and make the two communicate with each other. In the
instance of the class there are methods like forkHeadlessSqueak ,
forkHeadlessSqueakAndDo: etc.

So the functionality is there but since Mariano said he will be working to
improve OSProcess I thought that would a very useful area to work on. Since
not only will allows running on multiple cores but even to lift any memory
limits. Also I think this is a nice way to make the pharo enviroment which
I mean the running vm with an image and satelite files not only a
collection of objects but also a object itself  that can communicate with
other pharo enviroment instances (Pharo running processes) .

Fuel I assume enters here the equation as a data exchange format , the
problem I have with fuel is that its not backward compatible which for me
is a very big deal especially when we talk about threading and processes
which is a pretty fundamental feature. We need something that will be
stable but also code orientated that will remain backward compatible
because the last thing you want is initiating multiple processes that cant
understand each other because they happen to use diffirent image versions
that use diffirent verrsions of Fuel or what else.

On Sun, Dec 20, 2015 at 1:46 AM David T. Lewis  wrote:

> Mariano,
>
> I am really happy to see this and I look forword to helping you in
> any way I can.
>
> Dimitris,
>
> It is really interesting that you should suggest this. I think it may
> be a bit off topic for Mariano's initial goals, but Mariano is the
> driving force behind Fuel, which is the serializer that I use for
> RemoteTask in OSProcess/CommandShell. This attempts to do much of
> what you suggest, using Fuel to return the results of processing
> executed in a forked image (in the unix sense of "forked", i.e.
> extremely memory efficient).
>
> So maybe if I can help Mariano with his project (which of course I
> plan to do anyway), then he might be able to help me turn RemoteTask
> into something reliable enough for general purpose use.
>
> Another nice connection here is that Eliot's Spur image model will
> make #forkSqueak (the primitive behind RemoteTask) more efficient
> in cases where the cooperating images do a lot of processing, because
> memory changes are more localized in the Spur memory model. In principle
> it should be possible to run very large object memories in 64-bit Spur,
> while forking cooperating images that consume very little additional
> real memory.
>
> Dave
>
>
> On Sat, Dec 19, 2015 at 01:33:07PM +, Dimitris Chloupis wrote:
> > I am very excited for such a project.
> >
> > I hope you manage to fix the stdout problem on macos.
> >
> > One thing I would like to see is an expansion of the ability of OSProcess
> > to launch multiple instances of pharo and make them communicate with each
> > other. Thats a way to have pharo running on multiple cores. Maybe
> establish
> > a framework of communication of images that will be based on the existing
> > design of the fork mechanism in Pharo.
> >
> > Python is doing something similar with multiprocessing module  which
> > launches multiple processes of cpython executable and makes them
> > communication with each other through a similar api to its threading
> > module. This way python takes advantage of multiple cores. Python threads
> > like pharo are green threads and pharo like python has a GIL like
> > mechanism.
> >
> > Taking into account that current computers have at least 4 cores, its
> not a
> > bad idea to speed up pharo like this at least  400% ;)
> >
> > On Sat, Dec 19, 2015 at 3:24 PM Mariano Martinez Peck <
> marianop...@gmail.com>
> > wrote:
> >
> > > Dear all,
> > >
> > > I am tremendously happy to announce you that Pharo Consortium will
> sponsor
> > > yet another development effort. In this particular case, it's my honor
> to
> > > carry on such an effort and I will be developing and improving a few
> things
> > > and ideas. All the contract and paperwork has already been done so it's
> > > time to start working.
> > >
> > > Regarding the developments itself, we discussed about these topics:
> > >
> > > 1) Experiment with a simpler yet more limited OSProcess alternative to
> > > execute OS commands.
> > > 2) Improving FileSystem in order to better deal with some POSIX stuff
> like
> > > symbolic links, unix file permissions, etc etc.
> > >
> > > I have already started with 1). OSProcess is super complete and it
> > > involves some packages (OSProcess / CommandShell) as well as some VM
> > > plugins (OSProcessPlugin, AIOPlugin). OSProcess provides lots of
> features
> > > (like forking the running image) but we would like to focus only in
> > > executing OS commands. In addition, OSProcess dates from quite some
> years
> > > ago when many of the current infrastructure features did not 

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Mariano Martinez Peck]

Hi Dimitri, Dave et all.

First let me do a little disclaimer. Regarding the developments itself of
this sponsorship, we discussed about the topics I said in the first email.
However, those are not written in stone and that's why I also asked for
feedback. As you know, Pharo has relatively little resources so those
should be well spent. If we decide to take a spin into a project X, it
doesn't mean other projects are not worth. It simply means that we think X
will be the most useful for the time being.

One of the final goals of the discusses topics, is to improve a bit the
infrastructure for using Pharo as scripting (imagine Coral). For such a
thing, we believe that improving FileSystem and OSProcess-like is a good
first step to improve the underlying infrastructure.

That being said, I of course like the idea of distributing processing
across multiple images. No wonder, my PhD tool (Marea) and related
sub-projects (Fuel, Ghost proxies, etc) have something to do with the
mentioned topic. Dave already mentioned about RemoteTask, which was
something I was about to mention. There is also Seamless project (now used
by Remote Debugger stuff) and quite other tools. Finally, I think this
project may have a big percentage that could be considered as "research".
So having a PhD / master student working on this would also be nice. In
fact, maybe Denis Kudriashov might work on that topic??? (he is now in RMOD
team).

Also, not that forking the image is not my main short-term goal. As said, I
will experiment first with a simple tool to execute OS commands that would
be FFI based, trying to make async operations, and that would help in most
cases. Maybe for those that are not supported, then fallback in OSProcess.
Note that Dave (OSProcess author) is very helpful with me and we are
discussing together about this.

Torsten, thanks about OS. I will check out that project. I started with
OSProcess and Limbo. So it's good to have another to check.

I will try to make the survey starting next week.





On Sun, Dec 20, 2015 at 2:17 PM, Dimitris Chloupis 
wrote:

> how the communication is happening via Fuel , does it save to a fuel file
> or it does it through memory some way ?
>
> On Sun, Dec 20, 2015 at 6:04 PM David T. Lewis 
> wrote:
>
>> On Sun, Dec 20, 2015 at 05:07:47PM +0800, Pierce Ng wrote:
>> > On Sun, Dec 20, 2015 at 08:56:43AM +, Dimitris Chloupis wrote:
>> > > Fuel I assume enters here the equation as a data exchange format , the
>> > > problem I have with fuel is that its not backward compatible which
>> for me
>> >
>>
>> Fuel works well in this case. Version compatibility is not an issue,
>> because
>> when you fork an image/VM the two cooperating images are essentially
>> identical.
>>
>> > Once you have a set of OS processes running Pharo how they talk to each
>> other
>> > is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC,
>> JSONRPC,
>> > msgpack, coordinate via Redis/SQLite/PostgreSQL, etc.
>>
>> Any of those would work, but Fuel is a good way to enable all kinds of
>> objects to be copied from one image to another through a stream. In the
>> case of RemoteTask, all of the results of a remote computation can be put
>> into a single result object (an array or a dictionary or whatever), and
>> that result can be passed back to the parent image by reading the single
>> object. So - fork a copy of the image, have it do some work, and read the
>> result object from a stream. You know you are done when you have read
>> exactly one object.
>>
>> Dave
>>
>>


-- 
Mariano
http://marianopeck.wordpress.com

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Dimitris Chloupis]

not really because as I said UnixProcess already has an implementation for
doing this, you dont need any of those protocols/libraries you mentioned,
it does not even need Fuel. It already works.

And the reason why I mentioned Python multiprocessing module is because
multiprocessing does not only implement a specific way of communication
between processes its also handle concurrency issues like data
synchronization and much the of messy problems that come with concurrency.

And this is the are of improvement, its not what kind of format the
communication is , as you said it should be up to yourself how to format
your date, maybe you want json, maybe fuel, maybe you have your object and
classes. The real focus here is true concurrency issues on may experience
because you will be having two image running alongside with no guarantess
when one finish on action and framework to synchronize data.

Obviously one can implement his own but I think Pharo would greatly benefit
to have an establish default framework , even a small one for OS-Process
concurency.

On Sun, Dec 20, 2015 at 11:09 AM Pierce Ng  wrote:

> On Sun, Dec 20, 2015 at 08:56:43AM +, Dimitris Chloupis wrote:
> > Fuel I assume enters here the equation as a data exchange format , the
> > problem I have with fuel is that its not backward compatible which for me
>
> Once you have a set of OS processes running Pharo how they talk to each
> other
> is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC,
> JSONRPC,
> msgpack, coordinate via Redis/SQLite/PostgreSQL, etc.
>
> Pierce
>
>

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[David T. Lewis]

On Sun, Dec 20, 2015 at 05:07:47PM +0800, Pierce Ng wrote:
> On Sun, Dec 20, 2015 at 08:56:43AM +, Dimitris Chloupis wrote:
> > Fuel I assume enters here the equation as a data exchange format , the
> > problem I have with fuel is that its not backward compatible which for me
> 

Fuel works well in this case. Version compatibility is not an issue, because
when you fork an image/VM the two cooperating images are essentially identical.

> Once you have a set of OS processes running Pharo how they talk to each other
> is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC, JSONRPC,
> msgpack, coordinate via Redis/SQLite/PostgreSQL, etc.

Any of those would work, but Fuel is a good way to enable all kinds of
objects to be copied from one image to another through a stream. In the
case of RemoteTask, all of the results of a remote computation can be put
into a single result object (an array or a dictionary or whatever), and
that result can be passed back to the parent image by reading the single
object. So - fork a copy of the image, have it do some work, and read the
result object from a stream. You know you are done when you have read
exactly one object.

Dave

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Dimitris Chloupis]

how the communication is happening via Fuel , does it save to a fuel file
or it does it through memory some way ?

On Sun, Dec 20, 2015 at 6:04 PM David T. Lewis  wrote:

> On Sun, Dec 20, 2015 at 05:07:47PM +0800, Pierce Ng wrote:
> > On Sun, Dec 20, 2015 at 08:56:43AM +, Dimitris Chloupis wrote:
> > > Fuel I assume enters here the equation as a data exchange format , the
> > > problem I have with fuel is that its not backward compatible which for
> me
> >
>
> Fuel works well in this case. Version compatibility is not an issue,
> because
> when you fork an image/VM the two cooperating images are essentially
> identical.
>
> > Once you have a set of OS processes running Pharo how they talk to each
> other
> > is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC,
> JSONRPC,
> > msgpack, coordinate via Redis/SQLite/PostgreSQL, etc.
>
> Any of those would work, but Fuel is a good way to enable all kinds of
> objects to be copied from one image to another through a stream. In the
> case of RemoteTask, all of the results of a remote computation can be put
> into a single result object (an array or a dictionary or whatever), and
> that result can be passed back to the parent image by reading the single
> object. So - fork a copy of the image, have it do some work, and read the
> result object from a stream. You know you are done when you have read
> exactly one object.
>
> Dave
>
>

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Dimitris Chloupis]

yeah I completely agree that any improvement is more than welcomed. Thanks
I will check out RemoteTask and Seamless :)

On Sun, Dec 20, 2015 at 8:55 PM Mariano Martinez Peck 
wrote:

> Hi Dimitri, Dave et all.
>
> First let me do a little disclaimer. Regarding the developments itself of
> this sponsorship, we discussed about the topics I said in the first email.
> However, those are not written in stone and that's why I also asked for
> feedback. As you know, Pharo has relatively little resources so those
> should be well spent. If we decide to take a spin into a project X, it
> doesn't mean other projects are not worth. It simply means that we think X
> will be the most useful for the time being.
>
> One of the final goals of the discusses topics, is to improve a bit the
> infrastructure for using Pharo as scripting (imagine Coral). For such a
> thing, we believe that improving FileSystem and OSProcess-like is a good
> first step to improve the underlying infrastructure.
>
> That being said, I of course like the idea of distributing processing
> across multiple images. No wonder, my PhD tool (Marea) and related
> sub-projects (Fuel, Ghost proxies, etc) have something to do with the
> mentioned topic. Dave already mentioned about RemoteTask, which was
> something I was about to mention. There is also Seamless project (now used
> by Remote Debugger stuff) and quite other tools. Finally, I think this
> project may have a big percentage that could be considered as "research".
> So having a PhD / master student working on this would also be nice. In
> fact, maybe Denis Kudriashov might work on that topic??? (he is now in
> RMOD team).
>
> Also, not that forking the image is not my main short-term goal. As said,
> I will experiment first with a simple tool to execute OS commands that
> would be FFI based, trying to make async operations, and that would help in
> most cases. Maybe for those that are not supported, then fallback in
> OSProcess. Note that Dave (OSProcess author) is very helpful with me and we
> are discussing together about this.
>
> Torsten, thanks about OS. I will check out that project. I started with
> OSProcess and Limbo. So it's good to have another to check.
>
> I will try to make the survey starting next week.
>
>
>
>
>
> On Sun, Dec 20, 2015 at 2:17 PM, Dimitris Chloupis 
> wrote:
>
>> how the communication is happening via Fuel , does it save to a fuel file
>> or it does it through memory some way ?
>>
>> On Sun, Dec 20, 2015 at 6:04 PM David T. Lewis 
>> wrote:
>>
>>> On Sun, Dec 20, 2015 at 05:07:47PM +0800, Pierce Ng wrote:
>>> > On Sun, Dec 20, 2015 at 08:56:43AM +, Dimitris Chloupis wrote:
>>> > > Fuel I assume enters here the equation as a data exchange format ,
>>> the
>>> > > problem I have with fuel is that its not backward compatible which
>>> for me
>>> >
>>>
>>> Fuel works well in this case. Version compatibility is not an issue,
>>> because
>>> when you fork an image/VM the two cooperating images are essentially
>>> identical.
>>>
>>> > Once you have a set of OS processes running Pharo how they talk to
>>> each other
>>> > is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC,
>>> JSONRPC,
>>> > msgpack, coordinate via Redis/SQLite/PostgreSQL, etc.
>>>
>>> Any of those would work, but Fuel is a good way to enable all kinds of
>>> objects to be copied from one image to another through a stream. In the
>>> case of RemoteTask, all of the results of a remote computation can be put
>>> into a single result object (an array or a dictionary or whatever), and
>>> that result can be passed back to the parent image by reading the single
>>> object. So - fork a copy of the image, have it do some work, and read the
>>> result object from a stream. You know you are done when you have read
>>> exactly one object.
>>>
>>> Dave
>>>
>>>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Thierry Goubier]

Hi Mariano,

Le 20/12/2015 19:54, Mariano Martinez Peck a écrit :

Hi Dimitri, Dave et all.

First let me do a little disclaimer. Regarding the developments itself
of this sponsorship, we discussed about the topics I said in the first
email. However, those are not written in stone and that's why I also
asked for feedback. As you know, Pharo has relatively little resources
so those should be well spent. If we decide to take a spin into a
project X, it doesn't mean other projects are not worth. It simply means
that we think X will be the most useful for the time being.

One of the final goals of the discusses topics, is to improve a bit the
infrastructure for using Pharo as scripting (imagine Coral). For such a
thing, we believe that improving FileSystem and OSProcess-like is a good
first step to improve the underlying infrastructure.


Would we have access then to a command-line oriented minimal Pharo image?


That being said, I of course like the idea of distributing processing
across multiple images. No wonder, my PhD tool (Marea) and related
sub-projects (Fuel, Ghost proxies, etc) have something to do with the
mentioned topic. Dave already mentioned about RemoteTask, which was
something I was about to mention. There is also Seamless project (now
used by Remote Debugger stuff) and quite other tools. Finally, I think
this project may have a big percentage that could be considered as
"research". So having a PhD / master student working on this would also
be nice. In fact, maybe Denis Kudriashov might work on that topic??? (he
is now in RMOD team).

Also, not that forking the image is not my main short-term goal. As
said, I will experiment first with a simple tool to execute OS commands
that would be FFI based, trying to make async operations, and that would
help in most cases. Maybe for those that are not supported, then
fallback in OSProcess. Note that Dave (OSProcess author) is very helpful
with me and we are discussing together about this.


My guess is that, at least on Linux/Unix, that FFI will be exactly the 
same as the one used/expressed by the OSProcess plugin, that is 
popen/system.


Now, given some of the difficulties seen when dealing with external 
processes, I'd be wary of the "async" side of things. Random failures 
seemingly linked to signal loss are certainly not something I'd like to 
debug.


Something interesting and powerful would be pseudo-tty support. It would 
allow a Pharo image to maintain an open connection to a shell, for 
example, reducing the cost and latency of launching external commands 
and maintaining a context.


Regards,

Thierry

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[David T. Lewis]

Mariano,

I am really happy to see this and I look forword to helping you in
any way I can.

Dimitris,

It is really interesting that you should suggest this. I think it may
be a bit off topic for Mariano's initial goals, but Mariano is the
driving force behind Fuel, which is the serializer that I use for
RemoteTask in OSProcess/CommandShell. This attempts to do much of
what you suggest, using Fuel to return the results of processing
executed in a forked image (in the unix sense of "forked", i.e.
extremely memory efficient).

So maybe if I can help Mariano with his project (which of course I
plan to do anyway), then he might be able to help me turn RemoteTask
into something reliable enough for general purpose use.

Another nice connection here is that Eliot's Spur image model will
make #forkSqueak (the primitive behind RemoteTask) more efficient
in cases where the cooperating images do a lot of processing, because
memory changes are more localized in the Spur memory model. In principle
it should be possible to run very large object memories in 64-bit Spur,
while forking cooperating images that consume very little additional
real memory.

Dave


On Sat, Dec 19, 2015 at 01:33:07PM +, Dimitris Chloupis wrote:
> I am very excited for such a project.
> 
> I hope you manage to fix the stdout problem on macos.
> 
> One thing I would like to see is an expansion of the ability of OSProcess
> to launch multiple instances of pharo and make them communicate with each
> other. Thats a way to have pharo running on multiple cores. Maybe establish
> a framework of communication of images that will be based on the existing
> design of the fork mechanism in Pharo.
> 
> Python is doing something similar with multiprocessing module  which
> launches multiple processes of cpython executable and makes them
> communication with each other through a similar api to its threading
> module. This way python takes advantage of multiple cores. Python threads
> like pharo are green threads and pharo like python has a GIL like
> mechanism.
> 
> Taking into account that current computers have at least 4 cores, its not a
> bad idea to speed up pharo like this at least  400% ;)
> 
> On Sat, Dec 19, 2015 at 3:24 PM Mariano Martinez Peck 
> wrote:
> 
> > Dear all,
> >
> > I am tremendously happy to announce you that Pharo Consortium will sponsor
> > yet another development effort. In this particular case, it's my honor to
> > carry on such an effort and I will be developing and improving a few things
> > and ideas. All the contract and paperwork has already been done so it's
> > time to start working.
> >
> > Regarding the developments itself, we discussed about these topics:
> >
> > 1) Experiment with a simpler yet more limited OSProcess alternative to
> > execute OS commands.
> > 2) Improving FileSystem in order to better deal with some POSIX stuff like
> > symbolic links, unix file permissions, etc etc.
> >
> > I have already started with 1). OSProcess is super complete and it
> > involves some packages (OSProcess / CommandShell) as well as some VM
> > plugins (OSProcessPlugin, AIOPlugin). OSProcess provides lots of features
> > (like forking the running image) but we would like to focus only in
> > executing OS commands. In addition, OSProcess dates from quite some years
> > ago when many of the current infrastructure features did not exist yet.
> >
> > So the idea is to think a simpler alternative to execute OS commands,
> > using new features such as threaded FFI (for example, for reading async
> > from pipes), pinned objects, etc. We want to use FFI as much as possible
> > rather than VM plugins.
> >
> > From the image side, we are thinking about an API which would look like a
> > builder-like API (FileSystem, XStream, etc).
> >
> > We have been discussing with David Lewis (OSProcess author) as well as
> > with Eliot Miranda, Esteban Lorenzano, Damien Pollet, Stephane Ducasse, etc
> > in order to agree in a project that would be worth doing (otherwise we
> > would continue using existing tools). And they have all been very kind with
> > me providing positive discussions.
> >
> > I will soon do a survey to measure the most common use cases of OSProcess
> > and related tools.
> >
> > If you have any thought, question, feedback, or whatever, please let us
> > know.
> >
> > Finally, note that I will be doing this project together with another
> > client's project (Quuve) and my life does not have much free time these
> > days...so please be patient!!!
> >
> > Best,
> >
> > --
> > Mariano
> > http://marianopeck.wordpress.com
> >

Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium Sponsored Development Effort

[Dimitris Chloupis]

I am very excited for such a project.

I hope you manage to fix the stdout problem on macos.

One thing I would like to see is an expansion of the ability of OSProcess
to launch multiple instances of pharo and make them communicate with each
other. Thats a way to have pharo running on multiple cores. Maybe establish
a framework of communication of images that will be based on the existing
design of the fork mechanism in Pharo.

Python is doing something similar with multiprocessing module  which
launches multiple processes of cpython executable and makes them
communication with each other through a similar api to its threading
module. This way python takes advantage of multiple cores. Python threads
like pharo are green threads and pharo like python has a GIL like
mechanism.

Taking into account that current computers have at least 4 cores, its not a
bad idea to speed up pharo like this at least  400% ;)

On Sat, Dec 19, 2015 at 3:24 PM Mariano Martinez Peck 
wrote:

> Dear all,
>
> I am tremendously happy to announce you that Pharo Consortium will sponsor
> yet another development effort. In this particular case, it's my honor to
> carry on such an effort and I will be developing and improving a few things
> and ideas. All the contract and paperwork has already been done so it's
> time to start working.
>
> Regarding the developments itself, we discussed about these topics:
>
> 1) Experiment with a simpler yet more limited OSProcess alternative to
> execute OS commands.
> 2) Improving FileSystem in order to better deal with some POSIX stuff like
> symbolic links, unix file permissions, etc etc.
>
> I have already started with 1). OSProcess is super complete and it
> involves some packages (OSProcess / CommandShell) as well as some VM
> plugins (OSProcessPlugin, AIOPlugin). OSProcess provides lots of features
> (like forking the running image) but we would like to focus only in
> executing OS commands. In addition, OSProcess dates from quite some years
> ago when many of the current infrastructure features did not exist yet.
>
> So the idea is to think a simpler alternative to execute OS commands,
> using new features such as threaded FFI (for example, for reading async
> from pipes), pinned objects, etc. We want to use FFI as much as possible
> rather than VM plugins.
>
> From the image side, we are thinking about an API which would look like a
> builder-like API (FileSystem, XStream, etc).
>
> We have been discussing with David Lewis (OSProcess author) as well as
> with Eliot Miranda, Esteban Lorenzano, Damien Pollet, Stephane Ducasse, etc
> in order to agree in a project that would be worth doing (otherwise we
> would continue using existing tools). And they have all been very kind with
> me providing positive discussions.
>
> I will soon do a survey to measure the most common use cases of OSProcess
> and related tools.
>
> If you have any thought, question, feedback, or whatever, please let us
> know.
>
> Finally, note that I will be doing this project together with another
> client's project (Quuve) and my life does not have much free time these
> days...so please be patient!!!
>
> Best,
>
> --
> Mariano
> http://marianopeck.wordpress.com
>