Re: Working on pygments

[Edward K. Ream]

On Tuesday, March 19, 2019 at 2:57:31 PM UTC-5, Edward K. Ream wrote:

Using the pygments syntax colorer would be an 
> important addition to Leo.  This is #568 
> . Imo, it should be 
> part of the last python 2 version of Leo.
>

Rev 5ae87f5c in the pygments branch demonstrates all essential aspects of 
using pygments!  This is a huge milestone.

*To do*

At present, the code uses pygments.style.get_style_by_name('default') to 
specify colors. Most of Leo's present syntax-coloring-related settings will 
likely disappear, to be replaced by much simpler settings.

The colorer must tell pygments how to color section references and @doc 
parts.  This should be straightforward.

The present code uses only the python lexer. Leo already has a framework to 
change lexers in the middle of body text, so generalizing the code should 
be straightforward.

*Summary*

The pygments branch now demonstrates the essentials of using pygments for 
syntax coloring.  The entire leo/modes directory will disappear, along with 
much of Leo's legacy syntax coloring code.

Significant work remains, but I foresee no real difficulties.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Leo will use Jupyter's coloring code

[Edward K. Ream]

On Wed, Mar 20, 2019 at 9:15 AM john lunzer wrote:

Can you comment you on the impact in performance using this code will have?
>

Good question.

Using pygments should greatly improve performance.  The pygments token
parsers use regex's, which should be substantially faster than Leo's
existing jedit pattern matchers. See the node "jedit.Pattern matchers" in
leoColorizer.py.

The loop the node "jedit.mainLoop" will become:

for token in Lexer().get_tokens(s):

which will be *much *faster than looping *character by character* over s
and then calling the jedit pattern matchers one by one.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

Re: ENB: Leo will use Jupyter's coloring code

[john lunzer]

Can you comment you on the impact in performance using this code will have?

On Wednesday, March 20, 2019 at 8:22:36 AM UTC-4, Edward K. Ream wrote:
>
> The Jupyter syntax coloring code 
> 
>  
> from Jupyter's qt_console package appears to be exactly what Leo needs to 
> use pygments for syntax coloring!  This is a much-needed breakthrough.  The 
> Jupyter code is an elegant solution to a difficult problem.
>
> *The problem*
>
> Here Leo's existing (new) PygmentsColorizer.mainLoop code, from Leo's 
> pygments branch:
>
> def mainLoop(self, n, s):
> '''Colorize a *single* line s, starting in state n.'''
> trace = True and not g.unitTesting
> from pygments.lexers import PythonLexer
> d = {
> 'Keyword': 'keyword1',
> 'Keyword.Namespace': 'keyword1',
> 'Literal.String.Doc': 'literal1',
> # A **full** docstring.
> 'Literal.String.Interpol': 'literal1',
> 'Literal.String.Single': 'literal1',
> 'Comment.Single': 'comment1',
> # 'xt': 'blank',
> 'Operator': 'operator',
> }
> i = 0
> for token in PythonLexer().get_tokens(s):
> kind, val = token
> j = i + len(val)
> kind = repr(kind).lstrip('Token.')
> tag = d.get(kind)
> if tag:
> self.setTag(tag, s, i, j)
> i = j
>
> This *almost* works, but alas, it does not (and *can not*) handle tokens 
> that span lines. The code is drowning in an inch of water.
>
> *The Solution*
>
> The PygmentsHighlighter (PH) class monkey-patches 
> RegexLexer.get_tokens_unprocessed to store the final lexer stack on the 
> tokens themselves.  This is clever, non-trivial code.  Leo will use copies 
> of the Jupyter code. The only changes will be to replace a few Jupyter 2/3 
> switches with g.isString and g.isPython3.  I'll add extra the Jupyter 
> copyright to all top-level nodes, so there can be no doubt where the code 
> comes from.
>
> Here is PH.highlightBlock:
>
> def highlightBlock(self, string):
> """ Highlight a block of text."""
> prev_data = self.currentBlock().previous().userData()
> if prev_data is not None:
> self._lexer._saved_state_stack = prev_data.syntax_stack
> elif hasattr(self._lexer, '_saved_state_stack'):
> del self._lexer._saved_state_stack
> # Lex the text using Pygments
> index = 0
> for token, text in self._lexer.get_tokens(string):
> length = len(text)
> self.setFormat(index, length, self._get_format(token))
> index += length
> if hasattr(self._lexer, '_saved_state_stack'):
> data = PygmentsBlockUserData(
> syntax_stack=self._lexer._saved_state_stack)
> self.currentBlock().setUserData(data)
> # Clean up for the next go-round.
> del self._lexer._saved_state_stack
>
> This great stuff.  It will form the basis of PygmentsColorizer.mainLoop in 
> Leo. It should be relatively straightforward to make this compatible with 
> Leo's settings.
>
> *Summary*
>
> Leo will use Jupyter's syntax coloring code from Jupyter's qt_console 
> package. This is a crucial step in making #568 
>  work.
>
> The code will be copied into Leo, virtually unchanged. I'll add extra 
> copyright messages to all copied top-level nodes.
>
> Leo's PygmentsColorizer.mainLoop method will adapt PH.highlightBlock to 
> handle Leo's existing syntax-coloring settings.  This should be relatively 
> straightforward.  It may even be possible to use pygments' (or Jupyter's) 
> styling mechanism to do this.
>
> Edward
>

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

ENB: Leo will use Jupyter's coloring code

[Edward K. Ream]

The Jupyter syntax coloring code 

 
from Jupyter's qt_console package appears to be exactly what Leo needs to 
use pygments for syntax coloring!  This is a much-needed breakthrough.  The 
Jupyter code is an elegant solution to a difficult problem.

*The problem*

Here Leo's existing (new) PygmentsColorizer.mainLoop code, from Leo's 
pygments branch:

def mainLoop(self, n, s):
'''Colorize a *single* line s, starting in state n.'''
trace = True and not g.unitTesting
from pygments.lexers import PythonLexer
d = {
'Keyword': 'keyword1',
'Keyword.Namespace': 'keyword1',
'Literal.String.Doc': 'literal1',
# A **full** docstring.
'Literal.String.Interpol': 'literal1',
'Literal.String.Single': 'literal1',
'Comment.Single': 'comment1',
# 'xt': 'blank',
'Operator': 'operator',
}
i = 0
for token in PythonLexer().get_tokens(s):
kind, val = token
j = i + len(val)
kind = repr(kind).lstrip('Token.')
tag = d.get(kind)
if tag:
self.setTag(tag, s, i, j)
i = j

This *almost* works, but alas, it does not (and *can not*) handle tokens 
that span lines. Basically, the code is drowning in an inch of water.

*The Solution*

The PygmentsHighlighter (PH) class monkey-patches 
RegexLexer.get_tokens_unprocessed to store the final lexer stack on the 
tokens themselves.  This is clever, non-trivial code.  Leo will use copies 
of the Jupyter code. The only changes will be to replace a few Jupyter 2/3 
switches with g.isString and g.isPython3.  I'll add extra the Jupyter 
copyright to all top-level nodes, so there can be no doubt where the code 
comes from.

Here is PH.highlightBlock:

def highlightBlock(self, string):
""" Highlight a block of text."""
prev_data = self.currentBlock().previous().userData()
if prev_data is not None:
self._lexer._saved_state_stack = prev_data.syntax_stack
elif hasattr(self._lexer, '_saved_state_stack'):
del self._lexer._saved_state_stack
# Lex the text using Pygments
index = 0
for token, text in self._lexer.get_tokens(string):
length = len(text)
self.setFormat(index, length, self._get_format(token))
index += length
if hasattr(self._lexer, '_saved_state_stack'):
data = PygmentsBlockUserData(
syntax_stack=self._lexer._saved_state_stack)
self.currentBlock().setUserData(data)
# Clean up for the next go-round.
del self._lexer._saved_state_stack

This great stuff.  It will form the basis of PygmentsColorizer.mainLoop in 
Leo. It should be relatively straightforward to make this compatible with 
Leo's settings.

*Summary*

Leo will use Jupyter's syntax coloring code from Jupyter's qt_console 
package. This is a crucial step in making #568 
 work.

The code will be copied into Leo, virtually unchanged. I'll add extra 
copyright messages to all copied top-level nodes.

Leo's PygmentsColorizer.mainLoop method will adapt PH.highlightBlock to 
handle Leo's existing syntax-coloring settings.  This should be relatively 
straightforward.  It may even be possible to use pygments' (or Jupyter's) 
styling mechanism to do this.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To post to this group, send email to leo-editor@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.