Re: Leo's real weaknesses

[Offray Vladimir Luna Cárdenas]

Hi,

Nice to see how this discussion is moving forward.

I share the idea of longer names. These 3 letters don't convey anything 
to me and I have to go always to the docs to remember what they mean (I 
think g if for gui, and c for commander and no idea about p, but I'm 
always unsure). That being said, I don't think is "bad style", is not 
just "my style" and I prefer longer evocative names, but also I think 
that Leo, as all artifacts, it's  a product of its context. Live coding 
and power tab completion features encourage longer names, but also the 
community and authors practices (as in Pharo/Smalltalk) and sometimes 
one looks his own code and thinks that the future self would write it, 
in a complete different fashion. At least that happens to me with 
Grafoscopio and is just reaching 3 years old, so I imagine how this can 
be amplified in a 20 years old project like Leo.


What is a big advantage of Leo is what it enables: this community and 
stuff you cannot do in any other software. This uniqueness is a profound 
source of inspiration, as I have said before, and some inspiration is 
found in the stuff that Leo doesn't have (live coding, simpler 
collaboration and file format or even longer names), which doesn't 
diminish in any way, Leo's history and community. Thanks for them.


Cheers,

Offray


On 11/01/17 08:02, Edward K. Ream wrote:
Recent criticisms of Leo and its architecture have been wildly off the 
mark.  The most ludicrous was the statement that 'c', 'g' and 'p' 
should have longer names.


Here, I'd like to discuss Leo's /real /weaknesses. Some have recently 
been exposed as I study pyzo's operation and code. I'll also explain 
how they arose and what might be done about them. With one exception, 
these have nothing whatever to do with Leo's classes or code.


*Collaboration difficulties*

Imo, this is Leo's main weakness in the world. It's not all that easy 
to share .leo files. Diffs don't work well with xml files. Converting 
from xml to (say) json, won't help.


Jupyter is far superior in this regard. The Jupyter project has had to 
handle the usual security considerations.  It appears that they have 
come to a reasonable approach. This is one reason why the "Jupyter in 
Leo" project seems important to me.


*Weak or missing basic editor features*

Pyzo shows just how much Leo's syntax coloring, code completion and 
debugging capabilities could be improved.  I intend to remedy these 
defects this year, basing my work on pyzo's brilliantly simple code.


*Dubious decisions*

Most of these happened long ago, for what seemed like good reasons at 
the time.


1. *Using wrapper classes for text editing*

The present Qt gui is, iirc, Leo's fourth gui.  Two preceded Leo in 
python.  Previously, the python version of Leo used the much weaker Tk 
gui.


The pyzo code shows just how big a price Leo has paid for the wrapper 
text abstraction layer.  Otoh, having a standard text widget greatly 
simplifies the curses gui plugin. Still, the text wrappers do 
complicate matters significantly.


2. *Commands are defined in the Commands class*

This class really should be called the Controller class, using MVC 
terminology.  
Anyway, defining defining commands in the c class made sense 
initially.  Commands are clearly not parts of Leo's (data) model, nor 
are they part of a view.


If I were to do things all over again, I would likely split 'c' into 
'c' and 'c.commands' components.  But that's not possible now because 
it would break existing code.  In my defense, the original (python) 
design happened long before Python had decorators.


In fact, it's not even possible to split leoCommands.py into separate 
modules. I've actually tried (twice) and given up quickly.  Mistakes 
would break Leo and there aren't nearly enough unit tests to catch 
those mistakes.


3. *Underestimating regular expressions*

Almost all of the g.skip or g.scan functions could be replaced by 
regular expressions. That would greatly simplify the code.  I haven't 
done this because there are bigger fish to fry.


The g.skip or g.scan functions could also be rewritten /using/ regular 
expressions.  But this /must/ be done safely.  I will reject any pull 
requests that do not contain strong unit tests that demonstrate 
/complete /compatibility with previous versions.


*Outlines and clones complicate everything behind the scenes*

This is unavoidable.  It's not a mistake and it's not the result of 
bad design or poor programming.  The flip side is that leo can do 
things no other IDE can do.


But this /is/ still a weakness of Leo.  Leo's code can be 
complicated.  For example, supporting an in-Leo debugger requires 
Leo's goto-line-number code.  No big deal: that code is encapsulated.  
But there remain off-by-one errors that have to be fixed.


So that's it.  I've confessed to you all of Leo's real weaknesses, as 
I see them.  Leo's behind-the-scenes co

Re: Leo's real weaknesses

[Offray Vladimir Luna Cárdenas]

Hi,

Some years ago I proposed something for Leo, combining two technologies: 
fossil and Yaml[1], for the same reasons exposed here: fossil a as 
easier, self contained technology for working with files and Yaml as a 
diff friendly format to express graphs (trees with clones). At that time 
the idea didn't advance, but I explored it in Grafoscopio using STON[2] 
instead of Yaml. Now I have a diff and human friendly format for 
Grafoscopio notebooks, that can express graphs. For example, the 
interactive notebook about Panama Papers at [3] is shown in a fossil 
repository at [4] and a diff with a previous version is on [5].


[1] 
http://markmail.org/search/?q=leo-editor%20offray%20yaml%20fossil#query:leo-editor%20offray%20yaml%20fossil+page:1+mid:t7xlg7mt5hqnldkn+state:results


[2] https://github.com/svenvc/ston/blob/master/ston-paper.md
[3] http://mutabit.com/offray/blog/en/entry/panama-papers-1
[4] http://mutabit.com/repos.fossil/panama-papers/artifact/cbfe8929edf64212
[5] 
http://mutabit.com/repos.fossil/panama-papers/fdiff?sbs=1&v1=255e79b046584a36&v2=cbfe8929edf64212


Some notes about my thesis that are in Leo have conflicts created by xml 
+ dropbox, that made the XML unreadable from Leo again, as you can see 
in this fossil repo at [6]. I will try to fix the file to make it 
loadable again in Leo.


[6] 
http://mutabit.com/repos.fossil/doctorado-offray/dir?ci=ab8052282f0847de&name=Tesis


This shows how XML is really a big weakness for Leo and other programs 
and that other simpler approaches for a diff/human friendly format and 
technologies for storing complex documents and enabling collaborative 
projects are available, as the Grafoscopio prototypes shown, but, 
unfortunately, XML and git have monopolized the developer's imagination. 
But maybe now we can rethink some core assumptions about technologies 
and formats behind Leo.


I would try something like this:

- Storing a simple Leo tree as a Yaml tree, including clones and see how 
far can we push the format, for example, when interaction with external 
files is represented in that tree (@file, @clean @auto).
- Storing Leo and these external files in a single .fossil repository 
and explore if having this technology as a simple backed for storing 
files and their history makes easier to work with the changes in such 
external files. At least in my preliminary explorations, having fossil 
compiled with JSON support[7][8] has enabled to query, from Grafoscopio, 
the fossil repository with a simple interface (I have not made any POST 
experiment yet). Anyway, I could imagine a single .fossil file that 
makes Leo store a tree and related files, so if any of them changes, the 
changes are compared against the repository to ease collaboration.


[7] http://fossil.wanderinghorse.net/repos/fwiki/index.cgi/wiki/README
[8] 
https://docs.google.com/document/d/1fXViveNhDbiXgCuE7QDXQOKeFzf2qNUkBEgiUvoqFN4/view


Hope this helps,

Cheers,

Offray

On 13/01/17 14:41, Edward K. Ream wrote:
On Fri, Jan 13, 2017 at 2:22 PM, vitalije > wrote:


Fossil OTOH, keeps all its data in just one file which is in fact
just sqlite3 database file.


​Interesting.

​> ​
If you are interested in those scripts I will try to find (or 
recreate) and share them.


​Of course I'm interested ;-)

EKR
--
You received this message because you are subscribed to the Google 
Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to leo-editor+unsubscr...@googlegroups.com 
.
To post to this group, send email to leo-editor@googlegroups.com 
.

Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Syntax coloring mystery solved!!

['Terry Brown' via leo-editor]

Congrats on finding the culprit.
Couple of things I'd like to see in Leo syntax highlighting, highlighting of 
trailing whitespace, and highlighting of 
https://en.wikipedia.org/wiki/Non-breaking_space and also of tab (\t).  To be 
quite honest I've only made one foray into the syntax highlighting code, some 
time ago, and really can't remember the details.  So don't know if this is hard 
or trivial.
Cheers -Terry
 
  From: Edward K. Ream 
 To: leo-editor  
 Sent: Friday, January 13, 2017 12:23 PM
 Subject: Syntax coloring mystery solved!!
   
tl;dr: Calling doc.markContentsDirty(0, len(p.b)) suffices to recolor a node! 
This should be done instead of calling highlighter.rehighlight explicitly.

I could never have found this bug if I hadn't seen how fast pyzo colorizing 
code was within pyzo itself compared to the same code within Leo. 

Having refresh-from-disk do the coloring correctly and quickly was the crucial 
lucky break.  Tracing showed that a big @edit node was being colorized 
correctly despite highlighter.rehighlight not being called.

After a bit of noodling (without beer) I saw that the body pane's QTextDocument 
must be repeatedly calling highlighter.highlightBlock. I verified this was so 
(without huge traces) by examining the before and after counts of 
highlighter.n_calls.  As hoped, these jumped, indicating that something 
(QTextDocument is the only candidate) was in control.

This is a lucky, lucky break on Friday the thirteenth ;-) It is likely that 
Leo's existing colorizers can be speeded up by 10x. If so, there will be no 
need for the big text hack. Crucially, Qt does not show the text until it has 
been fully colorized.

Furthermore, close study of the pyzo code shows that actual syntax coloring 
could now be deferred until idle time, thereby further improving performance.

Pyzo is full of superb code, and pyzo's colorer is no exception.  I'm not sure 
whether to adapt pyzo's code for all syntax coloring, but I'm tempted.  It 
would be a big project.

In the meantime, I can adapt two good ideas from the pyzo colorer:

1. Precomputing QTextFormat objects saves a lot of time. Leo's present 
colorizers do far too much work withing the main colorizing loop.

2. Using a special tag for unterminated strings is genius.  The three little 
dots are a great visual cue, and terminating the token at the end of the line 
means that adding a quote does not cascade to all following lines.  A great 
hack.

I'm glad this is behind us.  I was starting to doubt my sanity, and starting to 
imagine bugs that would never be found...

The way is now open for more pyzo integration.  I am seriously considering 
replacing Leo's custom text widget with pyzo's.  It a big project, but it will 
pay huge dividends.

Edward
-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.


   
 

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Leo's real weaknesses

[Edward K. Ream]

On Fri, Jan 13, 2017 at 2:22 PM, vitalije  wrote:

Fossil OTOH, keeps all its data in just one file which is in fact just
> sqlite3 database file.
>

​Interesting.

​> ​
If you are interested in those scripts I will try to find (or recreate) and
share them.

​Of course I'm interested ;-)

EKR

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Leo's real weaknesses

[vitalije]

I suppose it can be done with the git also. But in that case user would 
need to have git installed and also git would need to recreate '.git' 
subfolder inside temporary folder every time. Fossil OTOH, keeps all its 
data in just one file which is in fact just sqlite3 database file. One can 
access its content with sqlite3 python module. Fossil itself is just one 
executable file about 1.5 - 2Mb which can be downloaded by a python script 
if it is not already available in the system path. Fossil executable 
contains also a standalone web server and can communicate and synchronize 
data with remote repositories through HTTP or SSH.

Scripts do not depend that much on fossil. They just run fossil through 
subprocess module. All that is needed to replace fossil with git is to 
change command line arguments. I believe fossil and git have similar 
commands but they use different names. 

If you are interested in those scripts I will try to find (or recreate) and 
share them.
Vitalije

On Friday, January 13, 2017 at 5:31:48 PM UTC+1, Edward K. Ream wrote:
>
> On Fri, Jan 13, 2017 at 4:53 AM, vitalije 
> > wrote:
>
> Converting from xml to (say) json, won't help.
>>>
>> IMHO it is not necessarily true. I don't suggest changing Leo document 
>> format from xml to json or any other format. But there can be found a 
>> scheme of encoding a Leo document in such a way that its changes can be 
>> expressed in a human readable diffs.
>>
>
> ​This is very interesting.  Do you think your ideas/scripts can be adapted 
> to git?
>
> > I didn't go any further in developing that idea mostly because I 
> couldn't find a good way to show those choices to the user and to show 
> different versions of a node. 
>
> Yes, this a real problem. I'm not sure even perfect, high-level diffs do 
> much good if the user can't use them easily.
>
> Edward
>

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Syntax coloring mystery solved!!

[Edward K. Ream]

*tl;dr:* Calling doc.markContentsDirty(0, len(p.b)) suffices to recolor a 
node! This should be done *instead of* calling highlighter.rehighlight 
explicitly.

I could never have found this bug if I hadn't seen how fast pyzo colorizing 
code was within pyzo itself compared to the *same* code within Leo. 

Having refresh-from-disk do the coloring correctly and quickly was the 
crucial lucky break.  Tracing showed that a big @edit node was being 
colorized correctly despite highlighter.rehighlight *not *being called.

After a bit of noodling (without beer) I saw that the body pane's 
QTextDocument must be repeatedly calling highlighter.highlightBlock. I 
verified this was so (without huge traces) by examining the before and 
after counts of highlighter.n_calls.  As hoped, these jumped, indicating 
that something (QTextDocument is the only candidate) was in control.

This is a lucky, lucky break on Friday the thirteenth ;-) It is likely that 
Leo's existing colorizers can be speeded up by 10x. If so, there will be no 
need for the big text hack. Crucially, Qt does not show the text until it 
has been fully colorized.

Furthermore, close study of the pyzo code shows that actual syntax coloring 
could now be deferred until idle time, thereby further improving 
performance.

Pyzo is full of superb code, and pyzo's colorer is no exception.  I'm not 
sure whether to adapt pyzo's code for all syntax coloring, but I'm 
tempted.  It would be a big project.

In the meantime, I can adapt two good ideas from the pyzo colorer:

1. Precomputing QTextFormat objects saves a lot of time. Leo's present 
colorizers do far too much work withing the main colorizing loop.

2. Using a special tag for unterminated strings is genius.  The three 
little dots are a great visual cue, and terminating the token at the end of 
the line means that adding a quote does not cascade to all following 
lines.  A great hack.

I'm glad this is behind us.  I was starting to doubt my sanity, and 
starting to imagine bugs that would never be found...

The way is now open for more pyzo integration.  I am seriously considering 
replacing Leo's custom text widget with pyzo's.  It a big project, but it 
will pay huge dividends.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Leo's real weaknesses

[Edward K. Ream]

On Fri, Jan 13, 2017 at 4:53 AM, vitalije  wrote:

Converting from xml to (say) json, won't help.
>>
> IMHO it is not necessarily true. I don't suggest changing Leo document
> format from xml to json or any other format. But there can be found a
> scheme of encoding a Leo document in such a way that its changes can be
> expressed in a human readable diffs.
>

​This is very interesting.  Do you think your ideas/scripts can be adapted
to git?

> I didn't go any further in developing that idea mostly because I couldn't
find a good way to show those choices to the user and to show different
versions of a node.

Yes, this a real problem. I'm not sure even perfect, high-level diffs do
much good if the user can't use them easily.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Understanding Leo Code

[Edward K. Ream]

On Tuesday, January 10, 2017 at 5:46:51 PM UTC-5, rengel wrote:

Beyond that, there are many other small annoyances: 
>
- In __doc__ strings, Leo code often states the obvious. Every Python 
> programmer knows, that __init__ is the constructor.
>

Leo's cff command often finds matches in multiple ctors.  These supposedly 
redundant comments are a recent addition.  The docstrings are typically:

'''ctor for x class.'''

They allow me to see to what the class each *cloned *ctor belongs.

EKR

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Leo's real weaknesses

[vitalije]

>
> Converting from xml to (say) json, won't help.
>
> IMHO it is not necessarily true. I don't suggest changing Leo document 
format from xml to json or any other format. But there can be found a 
scheme of encoding a Leo document in such a way that its changes can be 
expressed in a human readable diffs. That scheme can be just an additional 
format. Its purpose is not to replace xml. 

First let us analyze what kind of changes can we made by editing any given 
Leo document.

   1. we can edit content of any body, which means a standard plain text 
   changes (adding a line, deleting a line or changing some part of a line). A 
   standard diff tools are good in dealing with those kind of changes.
   2. or we can rearrange nodes. A simple edit such as sorting children of 
   one node, can make a very hard to understand diffs. 
   
A few years ago I tried to combine Leo with fossil to 
make it possible to see how any particular Leo document evolved. A fossil 
is just one executable file which can be easily incorporated with the Leo 
itself, it doesn't need to be installed or configured to work. 

I have made a script that encodes entire Leo tree into bunch of files one 
per each node named after gnx, and one special file named __LEOTREE__. The 
__LEOTREE__ file contents was just a list of lines that describe 
connections between nodes. Each line started with the level of the node it 
represents followed by its gnx. The node files were encoded with the simple 
scheme. Headline was a first line, followed by the body content of the 
node. The script would generate all those files in a temporary folder and 
then invoke a `fossil commit` there. Fossil would analyze content of the 
folder and record all changes.

The other script I have made would invoke fossil to print a timeline with 
all recorded versions, and offer a user to choose one of them. After user 
makes a choice that particular version would be extracted and content of 
the any given node that existed in that version would be displayed. 

I didn't go any further in developing that idea mostly because I couldn't 
find a good way to show those choices to the user and to show different 
versions of a node. That was before a view rendered plugin appeared, (which 
I still don't use). I guess maybe it could be used to show diffs or 
different versions of a node.
Vitalije

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.