Re: At last I understand sockets and servers

[Edward K. Ream]

On Mon, Oct 22, 2018 at 10:29 PM Terry Brown  wrote:

I just tried starting a Docker environment I have based on
> conda/miniconda3 which is based on debian, and it starts in under a
> second.  It has tex-live / numpy / matplotlib (which pulls in a chunk
> of Qt), so it's probably similar in size to something that would run
> Leo.
>

Thanks for this, Terry.  I haven't forgotten what you said.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: At last I understand sockets and servers

[Terry Brown]

On Sun, 21 Oct 2018 21:20:33 -0500
"Edward K. Ream"  wrote:

> Hmm.  Is Leo as a web app is feasible? Leo's scripting features make
> it extremely dangerous. If run on a server, Leo becomes an
> unverifiable cgi script. This looks like a gotcha.  Am I missing
> something?

Sandboxing / encapsulation with Docker or some similar container system
seems like the easy way.  Containers are much lighter than VMs, so you
should be able to run many sessions, depending on hardware etc.

Docker containers start fast enough if you're running something
long-lived like a session on leoBridge.py or something that runs for
the same length of time as a typical Leo session.  I *think* they'd be
too slow to exist transiently and respond to single requests, although
maybe hot cache on a suitable machine could run that way.  But I'm
guessing you're looking for something more persistent, anyway.

I just tried starting a Docker environment I have based on
conda/miniconda3 which is based on debian, and it starts in under a
second.  It has tex-live / numpy / matplotlib (which pulls in a chunk
of Qt), so it's probably similar in size to something that would run
Leo.

Cheers -Terry

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: At last I understand sockets and servers

[Edward K. Ream]

On Sunday, October 21, 2018 at 9:20:43 PM UTC-5, Edward K. Ream wrote:
>
> On Sun, Oct 21, 2018 at 11:28 AM Terry Brown  
> wrote:
>
> I wonder if Cormac was thinking of frameworks like Django, but perhaps
>> more appropriate (lighter) for Leo, Flask?  
>
>
> All options are on the table, but see below.
>
> Also mod_http.py plugin allows execution of arbitrary Python code in a
>> running Leo, so there's that to look at too.
>>
>
> Hmm.  Is Leo as a web app is feasible? 
>

To answer my own question, the answer is yes, as discussed further in 
answer to in my reply to Cormac 
.  
Imo, we must follow the Jupyter model, in which the server runs on the 
user's own (possibly virtual?) machine. It's time to study how Jupyter 
handles security issues.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: At last I understand sockets and servers

[Edward K. Ream]

On Sunday, October 21, 2018 at 11:01:57 AM UTC-5, Offray Vladimir Luna 
Cárdenas wrote:

It's good to see the advances in the servers direction. The idea of a more 
> decoupled Leo with web interfaces and the idea of "outlining" services to 
> other systems and languages that some have proposed seems close.
>

Interesting idea.  Providing "outlining services" seems like another 
project.

I would avoid going into the Apache path and I would try to choose a 
> minimalist pure python web framework (bottle, flask, other used by Jupyter) 
> that can deal with the server/sockets core code functionality in Leo and 
> leave the scaling part to a server neutral back-end (Apache, Cherokee, 
> Lighttpd, Enginx, etc.)
>

Sounds reasonable.  In another answer in this thread I said we could assume 
that Leo would use something like Apache (server), but I want to make clear 
that all options are on the table.

When I was trying to implement ideas from Leo in Pharo/Smalltak I remember 
> focusing on the "outlining experience" first. I wanted to be able to 
> add/remove and move nodes that have a "header" and a "body" inside Pharo. 
> Sometimes, when I think in this idea of "outlining" services using a web 
> interface I think which is the minimal web library that provides me a tree 
> interface, for example Inspire Tree[1], and how I would connect it to the 
> tree representation I already have in Grafoscopio. Because I don't want to 
> have a lot of moving parts in different languages and programming paradigms 
> (something that already happens to Jupyter), I would like to create some 
> socket/service that connects the web interface with the Pharo backend. So 
> all the machinery behind trees is already there and the only "new thing" 
> I'm implementing is a way to decouple the GUI so it can be a web app.
>
> [1] http://www.inspire-tree.com/
>
> Hopefully, something like that can be the path for services/sockets in 
> Leo. I wonder how Leo-Vue can connect to Leo server, for example and what 
> is the demo MVP for a web interface that connect with a Leo backend.
>

I don't know enough to respond in detail just now.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: At last I understand sockets and servers

[Edward K. Ream]

On Sunday, October 21, 2018 at 10:37:32 AM UTC-5, Cormac Williams wrote:

> I would answer some of your questions with more questions, mainly.

All these questions are reasonable.  Here are some preliminary answers. 

> 1 : Do you want to hang this server on the internet at some point?

Not sure what you mean by this.  Servers may spawn a new thread for each 
incoming thread.  In any case, we can assume that we'll use a fully-scaled 
server like Apache.

> 2 : What sort of scale do you want in terms of users?

Imo, n == 1 is often going to be the limit, for security reasons.  Leo 
looks like an unverifiable cgi script to the server, which means one user 
(or small, *trusted *group of users) must be *fully* responsible for the 
damage Leo could cause.

It might be possible to host a Leo server in a per-user (or per-small 
group) virtual machine somewhere, but that's it.  I see no way to run a 
public, unsecured, Leo server.

> 3 : What sort of scale do you want in terms of Leo outline size?

The same scale as at present.  This means that loading a Leo view in a 
browser won't load the entire xml file, but will only load the parents of 
presently-selected node.  In other words, Leo will use ajax to display the 
visible part of the outline.

> 4 : Do you want to get a database involved rather than serving from an 
xml file?

Good question. 

I recently thought about this in "plain" Leo.  There have been repeated 
requests for "node history".  One could imagine a DB containing old 
versions.  But I just recently saw that *node history would not have commit 
messages*.  Without those messages the "raw stream" of node contents is 
useless.  If the user wants node history, they should use git and be 
careful to write meaningful commit messages.  There is no way for Leo to 
provide those messages.

There is another reason for considering using a DB, namely to support 
cross-file clones.  The likely fatal problem with this is that the entire 
DB becomes a kind of cache.  External files must take precedence.  That 
being so, the DB adds a significant level of complexity, without 
fundamentally solving the problem.  This isn't likely to happen.

The third reason to use a DB would be to support huge outlines.  There may 
already be code in Leo to do this.  I don't remember.

> 5 : I there a multi-user story to be told in terms of using an outline?

See the answer to question 2.  The answer is yes, but only if all users 
bear full responsibility for the machine (virtual or not) on which Leo's 
server is run.

> 6 : How much are you prepared to refactor or change code to support the 
above visions?

I am willing to treat the project as essentially a new gui:  leo 
--gui=browser.

I will reject any suggestion to fork Leo's code base.  That would be the 
end of Leo, imo.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: At last I understand sockets and servers

[Edward K. Ream]

On Sun, Oct 21, 2018 at 11:28 AM Terry Brown  wrote:

I wonder if Cormac was thinking of frameworks like Django, but perhaps
> more appropriate (lighter) for Leo, Flask?


All options are on the table, but see below.

Also mod_http.py plugin allows execution of arbitrary Python code in a
> running Leo, so there's that to look at too.
>

Hmm.  Is Leo as a web app is feasible? Leo's scripting features make it
extremely dangerous. If run on a server, Leo becomes an unverifiable cgi
script. This looks like a gotcha.  Am I missing something?

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: At last I understand sockets and servers

[Cormac Williams]

"Edward K. Ream" > wrote: 

> > We can start by assuming ajax 
> > , using 
> > http. 
>
> Sounds a good way of communicating :-)

Terry wrote: 

> I wonder if Cormac was thinking of frameworks like Django, but perhaps 
> more appropriate (lighter) for Leo, Flask?  
>
> Cheers -Terry 
>

I was concentrating less on the actual technology to be used, but rather on 
the questions that I would ask before you start talking about the 
technology.

For example, I'm sure there are people out there serving leo files from 
FTP, cloud disks etc. Whilst technically that is client / server, 
effectively there is little additional benefit from doing so apart from 
files available at a distance.

On the other hand, there is the example of a web app like google docs 
spreadsheet, where multiple people can work on the sheet at the same time, 
it an be scaled to arbitrary sizes in terms of data and load etc. Code can 
be written to run server side as well as client side.

This thread was posted at the same time as I was exploring the leo code 
looking to answer the question :- "What is the smallest amount of changes 
necessary so as to read and write outlines from an ArangoDB database" :-)

Regards,
Cormac.

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: At last I understand sockets and servers

[Terry Brown]

On Sun, 21 Oct 2018 10:44:21 -0500
"Edward K. Ream"  wrote:

> On Sun, Oct 21, 2018 at 10:37 AM Cormac Williams
>  wrote:
> 
> > 
> 
> Hi.  Welcome to the discussion.
> 
> > Sockets and protocols are great, except if you want to hang it on
> > the internet, in which case you are far better using tried and
> > tested methodologies and frameworks.
> 
> I didn't mean to imply that sockets are all there are to the internet.
> Presumably a mature server like Apache will be required.
> 
> > Actually, I'd say use tried and tested for even local service. A
> > specialist protocol is only really required if you are doing
> > something radically different from what has come before, which I
> > get the feeling we are not talking about here.
> 
> We can start by assuming ajax
> , using
> http.

I wonder if Cormac was thinking of frameworks like Django, but perhaps
more appropriate (lighter) for Leo, Flask?  Django and Flask have
sub-components that handle translation between URL and Python code, and
I think there're libraries which *just* do that.  Flask's extra
features may or may not be useful.  Also being aware of WSGI is probably
good, but perhaps still a level lower than you need to be.

Also mod_http.py plugin allows execution of arbitrary Python code in a
running Leo, so there's that to look at too.

I've seen setup using Apache at the very front of a web-app hosting
ecosystem, with nginx (another web server) behind that handling
requests to actual backend services in Python (maybe by WSGI, not
necessarily).  At some level on all these systems though Python ends up
handling the request, so for development at least, you usually see
Python serving the web part.

Cheers -Terry


-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: At last I understand sockets and servers

[Offray Vladimir Luna Cárdenas]

Hi,

It's good to see the advances in the servers direction. The idea of a
more decoupled Leo with web interfaces and the idea of "outlining"
services to other systems and languages that some have proposed seems close.

I would avoid going into the Apache path and I would try to choose a
minimalist pure python web framework (bottle, flask, other used by
Jupyter) that can deal with the server/sockets core code functionality
in Leo and leave the scaling part to a server neutral back-end (Apache,
Cherokee, Lighttpd, Enginx, etc.)

When I was trying to implement ideas from Leo in Pharo/Smalltak I
remember focusing on the "outlining experience" first. I wanted to be
able to add/remove and move nodes that have a "header" and a "body"
inside Pharo. Sometimes, when I think in this idea of "outlining"
services using a web interface I think which is the minimal web library
that provides me a tree interface, for example Inspire Tree[1], and how
I would connect it to the tree representation I already have in
Grafoscopio. Because I don't want to have a lot of moving parts in
different languages and programming paradigms (something that already
happens to Jupyter), I would like to create some socket/service that
connects the web interface with the Pharo backend. So all the machinery
behind trees is already there and the only "new thing" I'm implementing
is a way to decouple the GUI so it can be a web app.

[1] http://www.inspire-tree.com/

Hopefully, something like that can be the path for services/sockets in
Leo. I wonder how Leo-Vue can connect to Leo server, for example and
what is the demo MVP for a web interface that connect with a Leo backend.

Cheers,

Offray


On 10/21/18 5:46 AM, Edward K. Ream wrote:
> The biggest roadblock of my career has disappeared. At last I see that:
>
>    *Sockets, not servers, are the basis of IPC (inter-process
> communication)
> *
>
> *   Servers exist because of sockets, not the other way around.
> *
>
> Getting this backward created no end of confusion. Everything seems
> clearer:
>
> - Connecting two processes means creating a socket between them.
>
> - Python's server (http.server in 3), asyncore and asynchat modules
> are just wrappers around sockets.
>
> - The request/response and publish/subscribe messaging patterns are
> two ways of using sockets. 
>
> - A server is required for /every /socket that uses the
> request/response pattern.
>
> - curl creates traffic on a socket, which a server then may handle, or
> not.
>
> *History*
>
> 1. All this became clear sometime between Friday morning and Saturday
> night, but I do not remember exactly when! It probably happened while
> talking to my brother Speed Friday afternoon.  I took these notes
> during that conversation:
>
> - Sockets are about IPC that "just works".  Pipes and shared memory
> are alternatives on a single machine, but sockets are essential for
> the internet.
>
> - Sockets are protocol agnostic. Sockets can use any protocol,
> including custom protocols.
>
> - The internet is a network of socket servers.  If I had to guess, it
> was this statement that triggered the Ahas. I immediately told Speed
> that this statement changed my understanding of the internet.
>
> 2. Yesterday morning I created my first socket using Python 3, created
> a small Python server based on this python 2 code
> , and used curl to
> test that server. That was the first time I ever used curl.
>
> 3.  Yesterday afternoon I talked with Speed again, discussing the Ahas
> in detail. I am fairly confident that I have the big picture right.
>
> 4. We then discussed Leo as a web app. Speed suggested that we avoid
> doing things twice. Python servers have their limitations, so
> eventually we will want to use a server like Apache.  That being so, I
> may as well start using Apache now.  That eliminates duplicating
> server configuration work.
>
> *Summary*
>
> Sockets, not servers, are the foundation almost all IPC, including the
> internet. This page
> 
> is a clear introduction to sockets.
>
> Focusing on sockets ended massive confusion about clients and servers.
>
> All internet software, including curl, handles traffic on sockets.
>
> Leo as a web app will use a server that responds to events in the
> browser. The socket protocol will likely be a private protocol rather
> than http. The server will be something like Apache, not a python server.
>
> All my old fears of internet-related technology are gone. Many thanks
> to Speed for his patient help. I eagerly look forward to lots of study.
>
> All comments, suggestions for further study, and (especially!)
> corrections are welcome. 
>
> Edward
> -- 
> You received this message because you are subscribed to the Google
> Groups "leo-editor" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to leo-editor+unsubscr...@googlegroups.com
> 

Re: At last I understand sockets and servers

[Edward K. Ream]

On Sun, Oct 21, 2018 at 10:37 AM Cormac Williams 
wrote:

> 
>

Hi.  Welcome to the discussion.

> Sockets and protocols are great, except if you want to hang it on the
> internet, in which case you are far better using tried and tested
> methodologies and frameworks.
>

I didn't mean to imply that sockets are all there are to the internet.
Presumably a mature server like Apache will be required.


> Actually, I'd say use tried and tested for even local service. A
> specialist protocol is only really required if you are doing something
> radically different from what has come before, which I get the feeling we
> are not talking about here.
>

We can start by assuming ajax
, using http.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: At last I understand sockets and servers

[Cormac Williams]

Tap, tap, tap...Hi everyone,

Been  lurking here a while, relatively new Leo user after having bounced 
off it a few times over the last few years.

I would answer some of your questions with more questions, mainly.

1 : Do you want to hang this server on the internet at some point?
2 : What sort of scale do you want in terms of users?
3 : What sort of scale do you want in terms of Leo outline size?
4 : Do you want to get a database involved rather than serving from an xml 
file?
5 : I there a multi-user story to be told in terms of using an outline? 
6 : How much are you prepared to refactor or change code to support the 
above visions?

Sockets and protocols are great, except if you want to hang it on the 
internet, in which case you are far better using tried and tested 
methodologies and frameworks. Actually, I'd say use tried and tested for 
even local service. A specialist protocol is only really required if you 
are doing something radically different from what has come before, which I 
get the feeling we are not talking about here.

Regards,
Cormac.


-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

At last I understand sockets and servers

[Edward K. Ream]

The biggest roadblock of my career has disappeared. At last I see that:

   
*Sockets, not servers, are the basis of IPC (inter-process communication)*


*   Servers exist because of sockets, not the other way around.*

Getting this backward created no end of confusion. Everything seems clearer:

- Connecting two processes means creating a socket between them.

- Python's server (http.server in 3), asyncore and asynchat modules are 
just wrappers around sockets.

- The request/response and publish/subscribe messaging patterns are two 
ways of using sockets.  

- A server is required for *every *socket that uses the request/response 
pattern.

- curl creates traffic on a socket, which a server then may handle, or not.

*History*

1. All this became clear sometime between Friday morning and Saturday 
night, but I do not remember exactly when! It probably happened while 
talking to my brother Speed Friday afternoon.  I took these notes during 
that conversation:

- Sockets are about IPC that "just works".  Pipes and shared memory are 
alternatives on a single machine, but sockets are essential for the 
internet.

- Sockets are protocol agnostic. Sockets can use any protocol, including 
custom protocols.

- The internet is a network of socket servers.  If I had to guess, it was 
this statement that triggered the Ahas. I immediately told Speed that this 
statement changed my understanding of the internet. 

2. Yesterday morning I created my first socket using Python 3, created a 
small Python server based on this python 2 code 
, and used curl to test 
that server. That was the first time I ever used curl.

3.  Yesterday afternoon I talked with Speed again, discussing the Ahas in 
detail. I am fairly confident that I have the big picture right.

4. We then discussed Leo as a web app. Speed suggested that we avoid doing 
things twice. Python servers have their limitations, so eventually we will 
want to use a server like Apache.  That being so, I may as well start using 
Apache now.  That eliminates duplicating server configuration work.

*Summary*

Sockets, not servers, are the foundation almost all IPC, including the 
internet. This page 
 
is a clear introduction to sockets.

Focusing on sockets ended massive confusion about clients and servers.

All internet software, including curl, handles traffic on sockets.

Leo as a web app will use a server that responds to events in the browser. 
The socket protocol will likely be a private protocol rather than http. The 
server will be something like Apache, not a python server.

All my old fears of internet-related technology are gone. Many thanks to 
Speed for his patient help. I eagerly look forward to lots of study.

All comments, suggestions for further study, and (especially!) corrections 
are welcome.  

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.