On Wed, Sep 21, 2016 at 11:11 AM, 'tfer' via leo-editor <
There are always request for new importers or changes to existing
> importers, few people other than Edward have wrapped their mind around how
> to do this for themselves. This is a proposal to make this easier for
> people to create their own importers or easily customize any importer to
> create nodes at whatever level of detail they want.
This a very long reply, but I want to explain exactly why the present code
is as it is.
This would work by borrowing the Unix principles of making little programs
> that do one thing, then composing a chain of them to accomplish the task
> you want done.
I am aware of this design pattern. The present code uses a different
pattern, namely having individual importers/writers override base importer
and writer classes.
The writer base class is relatively simple. Writing is much simpler than
The importer base class is complex for three distinct reasons:
1. Parsing is inherently a per-language process. We could use a different
parsing tool for each language, but that will lead to duplicate code. I
discuss parsing in greater detail below.
2. Given a proper parse of the language, splitting the code into separate
Leo nodes (what the importer calls code generation) is a tricky process.
We want to preserve line breaks and whitespace wherever possible.
3. We typically want to verify that the result of import will produce (when
written) the original import. There is a separate (*extremely complex*)
phase that does this, based on several switches in the overridden importer
So yes, one could turn each of these areas of code into a separate process,
but the actual code would not change much.
Take a look at CScanner class in leo/plugins/importers/c.py. It consists
only of a ctor that sets various switches. All the real work is done in
leo/plugins/importers/basescanner.py. There is *no way* to make the C
Imo, your proposal amounts to a request to refactor basescanner.py.
Perhaps that could be done, but I don't see any advantage to doing so.
Each language would have its own default chain, but you would have the
> option of adding your own chains/parameters/programs in your files
> "settings" node for each language you want to override the defaults on.
That's exactly the situation at present. Each importer uses settings to
modify the operation of basescanner.py, but importers are free to override
various methods as needed. For example, see importers/python.py.
Yes, the BaseScanner.Parsing methods are hairy. But there is a reason: the
parsing code, especially the scan and scanHelper must preserve line breaks
and whitespace. Traditional parsers don't do this. They simply create a
I have lots of experience with Python's AST parse trees. Annotating the
tree to show comments and whitespace is a big hole in the Python API. I
describe the workaround in this stack overflow page
As you can see, there are substantial difficulties involved.
But our problem is even harder: to create a *character-oriented* parser for
every imported language. The present parsing code works, though it is ugly
behind the scenes. Rather than using the base scanHelper method,
individual importers can override scanHelper. That's a feasible approach,
and it may be that some importers actually do that, but it doesn't change
the overall situation.
> I imagine some of these tools would be general and then made specific to
> the language at hand by passing them things like a keyword list, regex's,
> and the like. I'm not sure if this would need a intermediate format that
> gets turned into the nodes by a "nodeMaker" program at the end, (though
> this seems the likely scenario as the Unix approach is to keep things in
> text as long as possible), or if each program refines the parsing of nodes
> created by the previous program in the chain more directly.
The tools already exist. They are all the methods in basescanner.py.
Python has support for this sort of thing in the "included batteries", the
> io module includes stuff to work with strings. To make things look more
> Unix like their is pipetools: https://pypi.python.org/pypi/pipetools.
To summarize, your proposal amounts to a request to redesign or refactor
basescanner.py. I'm not going to do this, absent proof that the actual
problems to be solved are much easier than I know them to be :-)
You received this message because you are subscribed to the Google Groups
To unsubscribe from this group and stop receiving emails from it, send an email
To post to this group, send email to email@example.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.