Re: A new file format for Leo documents (sqlite3 database file)

[Terry Brown]

On Thu, 29 Jun 2017 14:49:43 -0700 (PDT)
vitalije  wrote:

> Ran 904 tests in 94.022s
> 
> FAILED (errors=6, skipped=25)

I had six errors testing master without modifications a couple of days
ago, so sounds like everything passed.  Well done.

Cheers -Terry

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: A new file format for Leo documents (sqlite3 database file)

[vitalije]

setting leoID from os.getenv('USER'): 'vitalije'
reading settings in 
/home/vitalije/programi/leo/trunk-git/leo/config/leoSettings.leo
reading settings in 
/home/vitalije/programi/leo/trunk-git/leo/config/myLeoSettings.leo
reading settings in 
/home/vitalije/programi/leo/trunk-git/leo/core/LeoPyRef.db

** isPython3: False
Leo 5.5, build 20170624181042, Sat Jun 24 18:10:42 CDT 2017
Git repo info: branch = sqlite-format, commit = 1e6da9e97999
Python 2.7.6, PyQt version 4.8.6
linux2
wrote recent file: /home/vitalije/.leo/.leoRecentFiles.txt
vitalije@phenom:~/programi/leo/trunk-git$ python launchLeo.py 
leo/test/unitTest.leo 
setting leoID from os.getenv('USER'): 'vitalije'
reading settings in 
/home/vitalije/programi/leo/trunk-git/leo/config/leoSettings.leo
reading settings in 
/home/vitalije/programi/leo/trunk-git/leo/config/myLeoSettings.leo
reading settings in 
/home/vitalije/programi/leo/trunk-git/leo/test/unitTest.leo

** isPython3: False
Leo 5.5, build 20170624181042, Sat Jun 24 18:10:42 CDT 2017
Git repo info: branch = sqlite-format, commit = 1e6da9e97999
Python 2.7.6, PyQt version 4.8.6
linux2
.saved: save-new-test.py
setting leoID from os.getenv('USER'): 'vitalije'
test of at.printError: La Peña
..s...
End of leoAtFile tests
..s
End of leoColor tests
.
End of leoCommands tests

End of leoConfig tests
..s..
.ss.s..s.ssssss...
End of typing tests
.
End of leoEditCommands tests.

End of leoFileCommands tests.
.
End of leoFind tests.
.
End of leoFrame tests.

End of leoGlobals tests.
.sss
End of leoGui tests.
...
End of leoImport tests.
...
End of leoKeys tests.

End of leoNodes tests.
ss..s..ss.EE...
End of leoUndo tests.
.
End of plugins unit tests
.@test print redraw count: 654
..all unit tests done
..
==
ERROR: runTest (leo.core.leoTest.GeneralTestCase)
@test rst3Test @no-head

--
Traceback (most recent call last):
  File "/home/vitalije/programi/leo/trunk-git/leo/core/leoTest.py", line 
211, in runTest
builtins.execfile(scriptFile, d)
  File "/home/vitalije/.leo/scriptFile.py", line 6, in 
rst3Test(c,p)
  File "", line 14, in __init__
  File "", line 54, in run
  File "", line 43, in clean
AttributeError: 'NoneType' object has no attribute 'replace'

==
ERROR: runTest (leo.core.leoTest.GeneralTestCase)
@test rst3Test default

--
Traceback (most recent call last):
  File "/home/vitalije/programi/leo/trunk-git/leo/core/leoTest.py", line 
211, in runTest
builtins.execfile(scriptFile, d)
  File "/home/vitalije/.leo/scriptFile.py", line 6, in 
rst3Test(c,p)
  File "", line 14, in __init__
  File "", line 54, in run
  File "", line 43, in clean
AttributeError: 'NoneType' object has no attribute 'replace'

==
ERROR: runTest (leo.core.leoTest.GeneralTestCase)
@test rst3Test doc_only_mode (set in headline)

--
Traceback (most recent call last):
  File "/home/vitalije/programi/leo/trunk-git/leo/core/leoTest.py", line 
211, in runTest
builtins.execfile(scriptFile, d)
  File "/home/vitalije/.leo/scriptFile.py", line 7, in 
rst3Test(c,p)
  File "", line 14, in __init__
  File "", line 54, in run
  File "", line 43, in clean
AttributeError: 'NoneType' object has no attribute 'replace'

==
ERROR: runTest (leo.core.leoTest.GeneralTestCase)
@test rst3Test doc_only_mode (set in options doc part)

--
Traceback (most recent call last):
  File "/home/vitalije/programi/leo/trunk-git/leo/core/leoTest.py", line 
211, in runTest
builtins.execfile(scriptFile, d)
  File "/home/vitalije/.leo/scriptFile.py", line 7, in 
rst3Test(c,p)
  File "", line 14, in __init__
  File "", line 54, in run
  File "", line 43, in clean
AttributeError: 

Re: A new file format for Leo documents (sqlite3 database file)

[Terry Brown]

On Thu, 29 Jun 2017 13:25:54 -0700 (PDT)
vitalije  wrote:

> In the other thread I have announced my experiment but it seems that
> I haven't explain it very well. Let me try another way.

Actually what you describe below is sort of what I thought you'd ended
up doing.  Sounds very interesting.

To run tests I 

python launchLeo.py leo/test/unitTest.leo

then press `Alt-4`  (not Alt-F4).  That file seems to be saved with the
right node selected.

As mentioned in another thread there are some tests that may fail on
system's other than Edward's.

Your file format supports p.v.u too?

Cheers -Terry

> The trouble I bumped into few days ago with changing shortcut
> settings was just a primer that set me in motion. My initial
> experiment with collecting all settings in sqlite database and speed
> test were intended just to make an argument. Although it started with
> the settings it was not only about settings. It was about new file
> format for Leo.  I knew from earlier discussion in this forum that
> Edward is/was not willing to approve such ideas. This time however I
> decided to give it a try just for myself and it happened that this
> time Edward actually encouraged me to go on with the experiment. But
> I suspect that my idea is still not very well explained/understood. 
> 
> There used to be somewhere on this forum some links to the talk that 
> Dr.Richard Hipp, the author of SQLite, gave explaining what he wish
> that user of SQLite library have in mind when using it. It is not
> meant to compete with multi-user, server based database systems. It
> is meant to replace a humble open (file ) function.
> 
> He gave some examples of missed opportunities  in some large software 
> projects like OpenOffice that use xml as file format for their
> documents. If they used SQLite instead they would be able to open
> document almost instantaneously, implementing undo/redo that can
> survive restart of application would be a trivial task, and lot of
> other advantages he has shown.  There are also several talks about
> the advantages of SQLite's source code control tool *fossil* over
> git. Not only that history (undo/redo) of one file is possible to be
> kept inside document file, but also the history of all project files,
> with all history details like check-in comments, wiki pages,
> branches, diffs,... all that stuff can be kept inside one file. User
> can copy that file on memory stick, send it via e-mail or google
> drive, it can be displayed online with just one executable file and
> it can be used for self hosting project site on the web. Something
> like github site packed in one executable file and one repository
> file. It handles user accounts, permissions, registrations, bug
> reports, to-dos,... almost everything that is needed for
> software-project hosting site. 
> 
> Inspired by those talks I have made a script that puts entire Leo
> tree in fossil repository and the other script that can recreate Leo
> tree from fossil repository. I remember it was very fast operation
> even if it used files on disk for the transfer to fossil. About as
> fast as is checking and parsing external files in Leo. Another script
> used to give a timeline list of all versions of Leo tree that are
> recorded in the repository. That list was printed in Log pane and it
> was possible for user to choose one of listed versions and Leo would
> recreate that particular version of tree.  
> 
> At that point I have abandoned the experiment because I haven't any
> clue how to create a GUI for all those features. It wasn't so useful
> with just a bunch of scripts and Log pane as the only GUI elements. I
> have also realized that there is a problem with the gnx-s. Two
> different versions of Leo tree had to be shown in different
> commanders because all changed nodes can not have the same gnx and
> different bodies or headlines. I still have a strong feeling that
> this idea of keeping the whole history of the tree in one file has
> some potential, but it needs suitable GUI to be useful at all. 
> 
> As I had no idea how suitable GUI could look like, nor how to solve
> the issue with conflicts between two different historical versions of
> the same node, I start to think about using sqlite database as file
> format for Leo documents. Knowing it would be hard to sell the idea,
> I let the time to pass and I did nothing about it. However that idea
> has never left my mind since. Recent events have lead me to final
> decision to do something about it. 
> 
> I have dived once again in Leo start-up code and this time I had more 
> success. I have finally made the first version of my prototype. With
> my changes Leo can accept as input argument database file. Now I can
> open any ordinary xml Leo document, click a button and database
> version of Leo document is generated in the same folder where Leo
> file is located and with the extension '.db' instead of '.leo'.
> Later Leo can open that db file and have the 

Re: A new file format for Leo documents (sqlite3 database file)

[Offray Vladimir Luna Cárdenas]

Congrats Vitalije!

Sometimes prototyping is the best way to argument in favor of an idea
and is nice to see your prototyping going well.

In my case, Grafoscopio [1] is a prototype of the ideas I would like to
see in Leo: liveness & interactivity, a simpler storage format (XML is
too cumbersome) and fossil integration. Grafoscopio documents are stored
in STON [2] (a Smalltalk extension of JSON) and versioned in a fossil
repository[3], so an outline includes all its history and the ones of
their related artefacts (data, images, other outlines) in a single
self-contained, self hosted fossil file. Browsing that history, merging
and diffing becomes trivial. Using Pharo have let me to explore this
ideas with agility and in a mostly solo endeavor as coder (but with the
important help of several communities).

[1] http://mutabit.com/grafoscopio/index.en.html
[2] https://github.com/svenvc/ston/blob/master/ston-paper.md
[3] http://mutabit.com/repos.fossil/grafoscopio/

I'm more interested in literate computing, interactive documentation and
reproducible research that in programming/scripting, so, while there is
certain overlap between Leo and Grafoscopio, they're looking for
different things, but is really nice to see common potent ideas being
developed, as the ones that is Vijalije sharing with us now.

Keep the good work and let us know how it goes,

Offray

On 29/06/17 15:25, vitalije wrote:
> In the other thread I have announced my experiment but it seems that I
> haven't explain it very well. Let me try another way.
>
> The trouble I bumped into few days ago with changing shortcut settings
> was just a primer that set me in motion. My initial experiment with
> collecting all settings in sqlite database and speed test were
> intended just to make an argument. Although it started with the
> settings it was not only about settings. It was about new file format
> for Leo.  I knew from earlier discussion in this forum that Edward
> is/was not willing to approve such ideas. This time however I decided
> to give it a try just for myself and it happened that this time Edward
> actually encouraged me to go on with the experiment. But I suspect
> that my idea is still not very well explained/understood. 
>
> There used to be somewhere on this forum some links to the talk that
> Dr.Richard Hipp, the author of SQLite, gave explaining what he wish
> that user of SQLite library have in mind when using it. It is not
> meant to compete with multi-user, server based database systems. It is
> meant to replace a humble open (file ) function.
>
> He gave some examples of missed opportunities  in some large software
> projects like OpenOffice that use xml as file format for their
> documents. If they used SQLite instead they would be able to open
> document almost instantaneously, implementing undo/redo that can
> survive restart of application would be a trivial task, and lot of
> other advantages he has shown.  There are also several talks about the
> advantages of SQLite's source code control tool *fossil* over git. Not
> only that history (undo/redo) of one file is possible to be kept
> inside document file, but also the history of all project files, with
> all history details like check-in comments, wiki pages, branches,
> diffs,... all that stuff can be kept inside one file. User can copy
> that file on memory stick, send it via e-mail or google drive, it can
> be displayed online with just one executable file and it can be used
> for self hosting project site on the web. Something like github site
> packed in one executable file and one repository file. It handles user
> accounts, permissions, registrations, bug reports, to-dos,... almost
> everything that is needed for software-project hosting site. 
>
> Inspired by those talks I have made a script that puts entire Leo tree
> in fossil repository and the other script that can recreate Leo tree
> from fossil repository. I remember it was very fast operation even if
> it used files on disk for the transfer to fossil. About as fast as is
> checking and parsing external files in Leo. Another script used to
> give a timeline list of all versions of Leo tree that are recorded in
> the repository. That list was printed in Log pane and it was possible
> for user to choose one of listed versions and Leo would recreate that
> particular version of tree.  
>
> At that point I have abandoned the experiment because I haven't any
> clue how to create a GUI for all those features. It wasn't so useful
> with just a bunch of scripts and Log pane as the only GUI elements. I
> have also realized that there is a problem with the gnx-s. Two
> different versions of Leo tree had to be shown in different commanders
> because all changed nodes can not have the same gnx and different
> bodies or headlines. I still have a strong feeling that this idea of
> keeping the whole history of the tree in one file has some potential,
> but it needs suitable GUI to be useful at all.  
>
> As I had no idea 

A new file format for Leo documents (sqlite3 database file)

[vitalije]

In the other thread I have announced my experiment but it seems that I 
haven't explain it very well. Let me try another way.

The trouble I bumped into few days ago with changing shortcut settings was 
just a primer that set me in motion. My initial experiment with collecting 
all settings in sqlite database and speed test were intended just to make 
an argument. Although it started with the settings it was not only about 
settings. It was about new file format for Leo.  I knew from earlier 
discussion in this forum that Edward is/was not willing to approve such 
ideas. This time however I decided to give it a try just for myself and it 
happened that this time Edward actually encouraged me to go on with the 
experiment. But I suspect that my idea is still not very well 
explained/understood. 

There used to be somewhere on this forum some links to the talk that 
Dr.Richard Hipp, the author of SQLite, gave explaining what he wish that 
user of SQLite library have in mind when using it. It is not meant to 
compete with multi-user, server based database systems. It is meant to 
replace a humble open (file ) function.

He gave some examples of missed opportunities  in some large software 
projects like OpenOffice that use xml as file format for their documents. 
If they used SQLite instead they would be able to open document almost 
instantaneously, implementing undo/redo that can survive restart of 
application would be a trivial task, and lot of other advantages he has 
shown.  There are also several talks about the advantages of SQLite's 
source code control tool *fossil* over git. Not only that history 
(undo/redo) of one file is possible to be kept inside document file, but 
also the history of all project files, with all history details like 
check-in comments, wiki pages, branches, diffs,... all that stuff can be 
kept inside one file. User can copy that file on memory stick, send it via 
e-mail or google drive, it can be displayed online with just one executable 
file and it can be used for self hosting project site on the web. Something 
like github site packed in one executable file and one repository file. It 
handles user accounts, permissions, registrations, bug reports, to-dos,... 
almost everything that is needed for software-project hosting site. 

Inspired by those talks I have made a script that puts entire Leo tree in 
fossil repository and the other script that can recreate Leo tree from 
fossil repository. I remember it was very fast operation even if it used 
files on disk for the transfer to fossil. About as fast as is checking and 
parsing external files in Leo. Another script used to give a timeline list 
of all versions of Leo tree that are recorded in the repository. That list 
was printed in Log pane and it was possible for user to choose one of 
listed versions and Leo would recreate that particular version of tree.  

At that point I have abandoned the experiment because I haven't any clue 
how to create a GUI for all those features. It wasn't so useful with just a 
bunch of scripts and Log pane as the only GUI elements. I have also 
realized that there is a problem with the gnx-s. Two different versions of 
Leo tree had to be shown in different commanders because all changed nodes 
can not have the same gnx and different bodies or headlines. I still have a 
strong feeling that this idea of keeping the whole history of the tree in 
one file has some potential, but it needs suitable GUI to be useful at all. 
 

As I had no idea how suitable GUI could look like, nor how to solve the 
issue with conflicts between two different historical versions of the same 
node, I start to think about using sqlite database as file format for Leo 
documents. Knowing it would be hard to sell the idea, I let the time to 
pass and I did nothing about it. However that idea has never left my mind 
since. Recent events have lead me to final decision to do something about 
it. 

I have dived once again in Leo start-up code and this time I had more 
success. I have finally made the first version of my prototype. With my 
changes Leo can accept as input argument database file. Now I can open any 
ordinary xml Leo document, click a button and database version of Leo 
document is generated in the same folder where Leo file is located and with 
the extension '.db' instead of '.leo'.  Later Leo can open that db file and 
have the same tree it would have if it parsed xml file. 

leoSettings and myLeoSettings as well as inside document settings are 
respected with the current implementation. 

The only thing currently missing is restoring the position and dimension of 
the application window on screen. 
It will be trivial to implement though.

I haven't run tests because I am not sure how to run them. Edward, Terry 
please give me the proper sequence of steps how to run all tests and check 
if I broke something or not by my changes to the code. If everything goes 
well I would publish git branch with my code 

Re: How good Leo handles settings?

[vitalije]

On Thursday, June 29, 2017 at 4:16:38 PM UTC+2, Terry Brown wrote:
>
> Hey Vitalije, what are you working on exactly?  Not wanting to 
> duplicate any effort and wondering if you're working on the sqlite 
> settings idea or something else, and if you have any thoughts on the 
> sqlite approach separating file specific settings from the file they 
> apply to, vs. the current system's inclusion of those settings in the 
> same file? 
>
> Cheers -Terr 
>

I am sorry for the delayed answer. I worked last night and came to the 
state where I have a script that can turn any open Leo document into sqlite 
database with the same name except the extension (for example: LeoPyRef.leo 
turns into LeoPyRef.db ). I have changed the isLeoFile so that it now 
accepts extensions .leo, .db and zip files. I have changed openLeoOrZipFile 
so that it checks if it is a .db file that needs to be opened and if so it 
opens a sqlite connection and returns it instead of open file. Returned 
value from this method goes unchanged through some helper methods and 
finally arrives in readSaxFile. There I have simulated the parsing of xml 
nodes to tree of SaxNodeClass objects and switched the call to 
fc.parse_leo_file which returns SaxNodeClass root node with the call to 
retrieve the tree of SaxNodeClass nodes from database. It allowed normal 
start of Leo and opening LeoPyRef.db as it were ordinary LeoPyRef.leo. 
Later I have discovered that it would be better to intercept the normal 
reading procedure not in readSaxFile but higher up in call chain in method 
getLeoFileHelper. Because with the first variant the values stored in 
database were lower level so that they represent tree of SaxNodeClass nodes 
which would then be transformed into v-nodes. Instead it is easier to put 
v-nodes directly in database and then to retrieve higher level v-nodes tree 
and set it to c.hiddenRootNode in getLeoFileHelper.

That was the end of my yesterday work. Today I couldn't touch it until now 
and now I am going to finish the prototype that is able to open any Leo 
file stored in sqlitedb and to read settings from the good old 
leoSettings.leo and myLeoSettings.leo files. 

Next I plan to make a script that would analyze all settings files and to 
put all settings into database table like in the previous experiment I did. 
The only thing missing in that experiment I believe was additional field in 
database table which should contain level of settings source. For example 
default value should be on level zero, leoSettings (if it is something 
different than default I am not sure) on level one, myLeoSettings on level 
2 and finally settings in leo document would have level 3. Then querying 
database about any settings value can be performed like this:

select value from settings where key=? order by level desc limit 1;
Which could be explained like give me the value of given key with the 
highest level.

The idea is that every Leo document would contain all settings defaults, 
leoSettings and myLeoSettings and also any additional settings for 
particular document. When user changes myLeoSettings settings, earlier 
created documents will have outdated values. In that case executing simple 
command would update all values inside db leo document.

To summarize:

   1. user can keep all present settings unchanged
   2. user still can change settings in myLeoSettings.leo
   3. every new created file would have those settings embedded
   4. user can choose to execute simple command to update settings in any 
   Leo db document or perhaps Leo could automatically check the modification 
   time of myLeoSettings.leo and offer a user to update settings if they are 
   outdated

Vitalije

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: How good Leo handles settings?

[Edward K. Ream]

On Thu, Jun 29, 2017 at 8:47 AM, Kent Tenney  wrote:

> >setting: long - primary key for settings table
> >name: text - setting name
> >body: text - text data for setting for docs. and @data settings
> >type: long - link to types table (@bool, @int, @data, etc.)
> >parent: long - link to settings table for hierarchy
> >outline: long - link to outline table
>

​Can you provide a link to the original message?  I don't see it.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: How good Leo handles settings?

[Terry Brown]

Hey Vitalije, what are you working on exactly?  Not wanting to
duplicate any effort and wondering if you're working on the sqlite
settings idea or something else, and if you have any thoughts on the
sqlite approach separating file specific settings from the file they
apply to, vs. the current system's inclusion of those settings in the
same file?

Cheers -Terr

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: How good Leo handles settings?

[Terry Brown]

On Thu, 29 Jun 2017 08:47:01 -0500
Kent Tenney  wrote:

> >setting: long - primary key for settings table
> >name: text - setting name
> >body: text - text data for setting for docs. and @data settings
> >type: long - link to types table (@bool, @int, @data, etc.)
> >parent: long - link to settings table for hierarchy
> >outline: long - link to outline table
> 
> Hows about a 'scope' field: one of
> default, global, user, curdir, fname

I think outline will cover that - if an outline is specified, the
setting applies to that outline.  If not, the search order for DB files
will be similar to that for (my)leoSettings.leo files.

But yes, the obvious implementation allows settings for files without
touching the file.  Trouble is I'm still not sure how to handle the
"file specific settings no longer in .leo file" issue - on the one hand
it seems no big deal that copying a .leo file to another system leaves
behind $HOME/myLeoSettings.leo, but having any settings that were in
the copied file (under the current system) not follow along seems like a
regression.

Cheers -Terry

> which are listed by increasing weight ...
> the 'fname' one would allow specifying settings for a file without
> touching it,
> which strikes me as kind of cool.
> 
> Maybe that's what is accomplished by the 'parent' and 'outline'
> fields, if so, please ignore this.
> 
> On Wed, Jun 28, 2017 at 3:55 PM, Terry Brown 
> wrote:
> 
> > On Wed, 28 Jun 2017 12:48:06 -0700 (PDT)
> > vitalije  wrote:
> >
> > > > ​I would be happy with that, provided that a) settings can be
> > > > organized
> > > >> and b) per-document (.leo file) settings are possible.​ ​ The
> > > >> best way of organizing anything is in a .leo file.​
> > > >
> > > As I wrote in the earlier post today, it is trivial in sqlite to
> > > open connection to several different databases (files). So it is
> > > easy to have one database file for settings machine wide, another
> > > one for settings per user, and yet another one for settings per
> > > folder (if I am not mistaken there used to be possible for user
> > > to define settings per project/folder, although I have never used
> > > that possibility). It is easy to make script to dump all relevant
> > > (leo, myLeo) settings-files in separate databases.
> >
> > The docs. on settings location are here:
> > http://leoeditor.com/customizing.html#id3
> >
> > not 100% sure but I assume `homeDir` is ~/.leo or $HOME\.leo and
> > `configDir` is .../leo/config/, and `localDir` is the folder
> > containing the .leo file.
> >
> > Hmm, I'd forgotten about the machine-name variation, although it
> > can be very useful when the same filesystem is accessed from
> > different hardware setups.
> >
> > Essential: 1a, 2b, 3
> > Consider dropping: 1b, 1c, 2a - really when would you use these?
> > Useful: 2c, 2d - 2d when you want .leo files in a folder to share
> >  settings
> >
> > So there's a question of when to use separate DB files vs. info. in
> > the DB to separate contexts.
> >
> > 1a and 2b would obviously be their own DB files, with a search that
> > fails in 2b being repeated in 1a, trusting for now that that won't
> > be a performance issue.
> >
> > 3 is trickier.  We're going to (ok, proposing to ;-) separate the
> > settings data from the rest of the .leo file(*).  Managing two files
> > side by side seems obnoxious, from the users point of view.  I
> > think it would be better to put the outline specific info. in the
> > same DB as used for 2b (e.g. ~/.leo/myLeoSettings.sqlite3).  So the
> > setting table might look like:
> >
> > setting: long - primary key for settings table
> > name: text - setting name
> > body: text - text data for setting for docs. and @data settings
> > type: long - link to types table (@bool, @int, @data, etc.)
> > parent: long - link to settings table for hierarchy
> > outline: long - link to outline table
> >
> > We'd start by looking at c.db to see how the outline table might
> > manage outline identities.
> >
> > (*) this separation is a high cost thing.  It means the user can't
> > just move the .leo file to another system and have that file's local
> > settings follow along.  This is consistent with myLeoSettings.leo
> > also not following along, but still seems like a regression.
> >
> > Which suggests that at least for file local settings we might want
> > to continue to support .leo file @settings trees.  Hmm.
> >
> > That leads to wanting to make .leo file @settings trees more
> > writable than they are now, which leads to the possibility of using
> > Leo outlines rather than sqlite as a storage format.  Before this
> > is interpreted as scraping the idea copletely, I'll point out the
> > leosax.py module which reads .leo files into a light weight dict
> > representation of an outline many many times faster than Leo's
> > regular read code.  It ignores external files, and is read only.
> >
> > Alternatives are
> 

Re: How good Leo handles settings?

[Kent Tenney]

>setting: long - primary key for settings table
>name: text - setting name
>body: text - text data for setting for docs. and @data settings
>type: long - link to types table (@bool, @int, @data, etc.)
>parent: long - link to settings table for hierarchy
>outline: long - link to outline table

Hows about a 'scope' field: one of
default, global, user, curdir, fname

which are listed by increasing weight ...
the 'fname' one would allow specifying settings for a file without touching
it,
which strikes me as kind of cool.

Maybe that's what is accomplished by the 'parent' and 'outline' fields, if
so, please ignore this.





On Wed, Jun 28, 2017 at 3:55 PM, Terry Brown  wrote:

> On Wed, 28 Jun 2017 12:48:06 -0700 (PDT)
> vitalije  wrote:
>
> > > ​I would be happy with that, provided that a) settings can be
> > > organized
> > >> and b) per-document (.leo file) settings are possible.​ ​ The best
> > >> way of organizing anything is in a .leo file.​
> > >
> > As I wrote in the earlier post today, it is trivial in sqlite to
> > open connection to several different databases (files). So it is easy
> > to have one database file for settings machine wide, another one for
> > settings per user, and yet another one for settings per folder (if I
> > am not mistaken there used to be possible for user to define settings
> > per project/folder, although I have never used that possibility). It
> > is easy to make script to dump all relevant (leo,
> > myLeo) settings-files in separate databases.
>
> The docs. on settings location are here:
> http://leoeditor.com/customizing.html#id3
>
> not 100% sure but I assume `homeDir` is ~/.leo or $HOME\.leo and
> `configDir` is .../leo/config/, and `localDir` is the folder containing
> the .leo file.
>
> Hmm, I'd forgotten about the machine-name variation, although it can be
> very useful when the same filesystem is accessed from different
> hardware setups.
>
> Essential: 1a, 2b, 3
> Consider dropping: 1b, 1c, 2a - really when would you use these?
> Useful: 2c, 2d - 2d when you want .leo files in a folder to share
>  settings
>
> So there's a question of when to use separate DB files vs. info. in the
> DB to separate contexts.
>
> 1a and 2b would obviously be their own DB files, with a search that
> fails in 2b being repeated in 1a, trusting for now that that won't be a
> performance issue.
>
> 3 is trickier.  We're going to (ok, proposing to ;-) separate the
> settings data from the rest of the .leo file(*).  Managing two files
> side by side seems obnoxious, from the users point of view.  I think it
> would be better to put the outline specific info. in the same DB as
> used for 2b (e.g. ~/.leo/myLeoSettings.sqlite3).  So the setting table
> might look like:
>
> setting: long - primary key for settings table
> name: text - setting name
> body: text - text data for setting for docs. and @data settings
> type: long - link to types table (@bool, @int, @data, etc.)
> parent: long - link to settings table for hierarchy
> outline: long - link to outline table
>
> We'd start by looking at c.db to see how the outline table might manage
> outline identities.
>
> (*) this separation is a high cost thing.  It means the user can't just
> move the .leo file to another system and have that file's local
> settings follow along.  This is consistent with myLeoSettings.leo also
> not following along, but still seems like a regression.
>
> Which suggests that at least for file local settings we might want to
> continue to support .leo file @settings trees.  Hmm.
>
> That leads to wanting to make .leo file @settings trees more writable
> than they are now, which leads to the possibility of using Leo outlines
> rather than sqlite as a storage format.  Before this is interpreted as
> scraping the idea completely, I'll point out the leosax.py module which
> reads .leo files into a light weight dict representation of an outline
> many many times faster than Leo's regular read code.  It ignores
> external files, and is read only.
>
> Alternatives are
>
> (a) keeping the .leo and settings DBs side by side,
> something.leo and something.leo.settings.sqlite?
> (b) embedding the sqlite data in a the .leo XML file -
> that seems like a horrible idea.
>
> So I think the question of the cost of separating file-local settings
> from the .leo file is an issue here.  It sort of runs contrary to the
> idea of XML, which should allow everything to be in one file.
>
> But anything that deviates from simple vanilla use of sqlite DB files
> starts to lose the simplicity gain.  Hmm.
>
> Cheers -Terry
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "leo-editor" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to leo-editor+unsubscr...@googlegroups.com.
> To post to this group, send email to leo-editor@googlegroups.com.
> Visit this group at 

Re: How good Leo handles settings?

[Edward K. Ream]

On Thursday, June 29, 2017 at 5:43:08 AM UTC-5, Edward K. Ream wrote:

> Or Leo could cache leoSettings.leo and myLeoSettings.leo, pretty much as 
described in #511 .
> I've just added caching to the list of items to be handled by the renamed 
> #510: 
improve how Leo handles settings 
.

Actually, Leo reads settings files only once, no matter how many outlines 
are loaded.  Yes, It would be slightly faster to cache settings files 
*between* invocations of Leo, but this would be a fairly minor speed 
improvement.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: How good Leo handles settings?

[Edward K. Ream]

On Thursday, June 29, 2017 at 4:57:04 AM UTC-5, Edward K. Ream wrote:
>
> On Wednesday, June 28, 2017 at 11:53:42 AM UTC-5, vitalije wrote:
>

So Leo opens *two* documents (the settings files) before opening ekr.leo, 
> and then Leo reads ekr.leo twice.
>
> Is it really necessary? It is if we insist on present scheme for dealing 
>> with settings. OTOH if Leo document was database it could contain all 
>> settings in one place and even if we want to have settings per machine/per 
>> user/ per folder and per document it can be done with no effort. We simply 
>> need to open and attach several databases. Sqlite allows that out of the 
>> box.
>>
>
> Or Leo could cache leoSettings.leo and myLeoSettings.leo, pretty much as 
> described in #511 . 
> The cache could even be a sqlite db.
>

I've just added caching to the list of items to be handled by the renamed #510: 
improve how Leo handles settings 
.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: I've stoped *saying* all tests pass in commit messages

[Edward K. Ream]

On Wed, Jun 28, 2017 at 11:26 AM, Terry Brown  wrote:

> Here are the fails I get from d1608d30.
>

​Thanks for this. I've just created #512
.

One fix would be to call self.skipTest in various​

​"difficult" tests unless I happen to be running them.

Presumably there is a reason why these tests fail for you but not for me.
It's time for me to actually load up my laptop.  I'll see whether the tests
fail there...

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: How good Leo handles settings?

[Edward K. Ream]

On Wednesday, June 28, 2017 at 11:53:42 AM UTC-5, vitalije wrote:

So Leo actually reads four Leo documents before it starts opening your 
> document and it read it twice. 
>

When I open ekr.leo, I see the following:

reading settings in C:\leo.repo\leo-editor\leo\config\leoSettings.leo
reading settings in C:\Users\edreamleo\.leo\myLeoSettings.leo
reading settings in C:\Users\edreamleo\ekr.leo

So Leo opens *two* documents (the settings files) before opening ekr.leo, 
and then Leo reads ekr.leo twice.

Is it really necessary? It is if we insist on present scheme for dealing 
> with settings. OTOH if Leo document was database it could contain all 
> settings in one place and even if we want to have settings per machine/per 
> user/ per folder and per document it can be done with no effort. We simply 
> need to open and attach several databases. Sqlite allows that out of the 
> box.
>

Or Leo could cache leoSettings.leo and myLeoSettings.leo, pretty much as 
described in #511 . 
The cache could even be a sqlite db.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: How good Leo handles settings?

[Edward K. Ream]

On Wed, Jun 28, 2017 at 6:09 PM, Edward K. Ream  wrote:

@Edward, I have another question. Where is the point where the actual
> reading of Leo document ends and data is transferred in the tree.
>
>  To be more concrete I have found in method getLeoFileHelper of
> FileCommands class the following lines:
>
> c.setRootVnode(v)
>
> fc.rootVnode = v
>
> I thought that is what I was looking for, but on my surprise
> c.setRootVnode is empty method. It does nothing.
>
> It looks like *both* the statements you show can be eliminated. Not only
> does c.setRootVnode do nothing, but fc.rootVnode is never used! I'll clean
> this up soon.
>

​Done at 259ae3e in master:

- Added a comment that readSaxFile sets c.hiddenRootNode.
- Replaced the two statements with a "pass".
- Removed the fc.rootVnode ivar entirely.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: Enhancement #511: reload-leo command

[Edward K. Ream]

On Thursday, June 29, 2017 at 4:15:38 AM UTC-5, Edward K. Ream wrote:
>
> #511: reload-leo command 
>  promises to make 
> Leo a killer app for python development.  It distills the back and forth in 
> this 
> conversation 
> .
>

I'd like to emphasize two points from #511: 

*The reload-leo command should be extremely fast*. Leo will read nothing 
from the file system. There will be no parsing of .leo files (no calls to 
sax) and no parsing of external files (no scanning for sentinel lines).

*There should never be any need to cache settings themselves*. Caching the 
settings *files *should suffice. The command will reload and parse the 
cached settings outlines almost instantaneously. There will be no special 
cases relating to settings. Users can change the settings files and use 
reload-leo as a complete alternative to reload-settings!

In other words, the recent discussion about settings is a completely 
separate matter.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Enhancement #511: reload-leo command

[Edward K. Ream]

#511: reload-leo command 
 promises to make Leo 
a killer app for python development.  It distills the back and forth in this 
conversation 
. 
reload-leo would complete/moot #417: complete reload-settings. 


As I write this, I realize the present proposal only reloads Leo itself 
(including settings).  Assuming this project succeeds, we would probably 
want something like a reload-app command, that would reload a given python 
app.  

Feel free to make comments here, or on the #511 page.

Edward


-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.