Re: fast-color branch now merged with the trunk
On Feb 10, 6:31 pm, Edward K. Ream edream...@gmail.com wrote: Rev 1538 of the trunk disables the colorizer if not hasattr(highlighter,'currentBlock') I have only tested that the code works if the highlighter *does* have the currentBlock method. Please verify that coloring is disabled if the highlighter doesn't have currentBlock. Did you push this change? I still get the same crash on hardy. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Thu, Feb 12, 2009 at 3:43 AM, Ville M. Vainio vivai...@gmail.com wrote: On Feb 10, 6:31 pm, Edward K. Ream edream...@gmail.com wrote: Rev 1538 of the trunk disables the colorizer if not hasattr(highlighter,'currentBlock') I have only tested that the code works if the highlighter *does* have the currentBlock method. Please verify that coloring is disabled if the highlighter doesn't have currentBlock. Did you push this change? Yes. Maybe the check is in the wrong place. Edward --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Tue, Feb 10, 2009 at 9:56 PM, Ville M. Vainio vivai...@gmail.com wrote: Haha. It looks like the hack of prepending the restart string to the text to be searched is going to be a problem. all_s[offset:] can be very large. True, in most cases restartString will almost always be empty, but a single worst case could crash Leo. That's not acceptable. I don't think this is a problem - it never saw up in ctrl+c logs. I disagree. I think it is essential to allocate a small number of strings. Imo, once the colorizer allocates on the order of 60,000 strings, each of the order of 100,000 bytes, it is going to encounter a massive slowdown. This, not any internals of QTextEdit, it the heart of the problem. Furthermore, reverting to the modified version of the old way (using indices) replaces all but one of those strings by a python dict containing indices. The dict will typically have only a few entries, and not much more than that for the worst case. The difference for the gc is huge and cannot be ignored. Edward --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Wed, Feb 11, 2009 at 2:54 PM, Edward K. Ream edream...@gmail.com wrote:
I disagree. I think it is essential to allocate a small number of strings.
Imo, once the colorizer allocates on the order of 60,000 strings, each of
the order of 100,000 bytes, it is going to encounter a massive slowdown.
This, not any internals of QTextEdit, it the heart of the problem.
Python is actually pretty fast at this kind of stuff.
This script completes in 1.1 seconds, and allocates qtGui.py contents
in a string1 times (qtgui has 8127 lines):
s = open('leo-editor/leo/plugins/qtGui.py').read()
for i in range(1):
s2 = hello + s
It's pretty bad (and sloppy), but not really the biggest problem in
this recolor. It took long enough that I tired of waiting...
--
Ville M. Vainio
http://tinyurl.com/vainio
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
leo-editor group.
To post to this group, send email to leo-editor@googlegroups.com
To unsubscribe from this group, send email to
leo-editor+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/leo-editor?hl=en
-~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Wed, Feb 11, 2009 at 9:23 AM, Ville M. Vainio vivai...@gmail.com wrote: On Wed, Feb 11, 2009 at 2:54 PM, Edward K. Ream edream...@gmail.com wrote: I disagree. I think it is essential to allocate a small number of strings. Imo, once the colorizer allocates on the order of 60,000 strings, each of the order of 100,000 bytes, it is going to encounter a massive slowdown. This, not any internals of QTextEdit, it the heart of the problem. Python is actually pretty fast at this kind of stuff. Rev 1546 of the fast-color branch now contains a synthesis of the older (big-state) and newer (string-state) approaches. States are bunches, not strings, but traces uses state names that are the same as the old string states. Furthermore, the code minimizes the number of states that are created. For example, at most two states were ever created for any node while running all unit tests. As usual, in the discussion below, w is a leoQtTextEditWidget. It was not possible to use onTextChanged because it gets called after incremental recoloring happens. Instead, the w.insert, w.delete and w.setAllText methods just set colorer.initFlag = True. The recolor method sets self.all_s if this flag is True (and then clears the flag). Thus, self.all_s will get set at most once per keystroke, and the issue of whether string allocation stresses the gc becomes moot. Likewise, the speed of w.getAllText is now moot, though I did call unicode(s) rather than g.toUnicode. Pasting qtGui.py into a node takes several seconds for the initial colorizing. At present, the recoloring happens twice, which obviously cries out for improvement. Thereafter, typing and coloring work tolerably well. Typing several characters in a row will queue the characters until you stop typing, at which they all get inserted and colorized at once. Traces show that only a few calls to recolor get made, so I don't think there is anything more that recolor can do. I'll eat my own dog food a bit longer before merging this into the trunk, but things are looking good. Edward --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Wed, Feb 11, 2009 at 7:22 PM, Edward K. Ream edream...@gmail.com wrote: It was not possible to use onTextChanged because it gets called after incremental recoloring happens. Instead, the w.insert, w.delete and Yeah, I thought you could *reset* the cache there. But either way is fine, I suppose. Pasting qtGui.py into a node takes several seconds for the initial colorizing. At present, the recoloring happens twice, which obviously cries out for improvement. Yeah. And at least it's fast enough that running it through profiler makes sense :-). Thereafter, typing and coloring work tolerably well. Typing several characters in a row will queue the characters until you stop typing, at which they all get inserted and colorized at once. Traces show that only a few calls to recolor get made, so I don't think there is anything more that recolor can do. Perhaps the problem is not recolor, but some other inefficiency somewhere. Well, profiler will tell. BTW, did ensure that @nocolor is handled in a very fast fashion? It's critical to skip all of this machinery for huge log files/data -- Ville M. Vainio http://tinyurl.com/vainio --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Feb 11, 11:22 am, Edward K. Ream edream...@gmail.com wrote: Rev 1546 of the fast-color branch now contains a synthesis of the older (big-state) and newer (string-state) approaches. States are bunches, not strings, but traces uses state names that are the same as the old string states. Furthermore, the code minimizes the number of states that are created. For example, at most two states were ever created for any node while running all unit tests. After so much work, I think we are finally at the best-possible place. The computeStateName method returns a state name used as a key into a dict whose keys are names and whose values are state numbers. This allows Leo to minimize the number of states created by the QSyntaxHighlighter. This, by itself, is a trivial optimization, but there is an important hidden property: state names are different **if and only if** the corresponding state bunches are different. As I said earlier, redrawing text only once when a node is selected is the last remaining task. It may be a bit tricky to do... Edward --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Wed, Feb 11, 2009 at 11:48 AM, Ville M. Vainio vivai...@gmail.comwrote: Yeah. And at least it's fast enough that running it through profiler makes sense :-). Here is a typical profiler run: Ordered by: file name, call count, internal time, function name List reduced from 2603 to 245 due to restriction 'qtGui.py:' ncalls tottime percall cumtime percall filename:lineno(function) 333070.6420.0002.3390.000 qtGui.py:6680(recolor) 333070.2090.0002.5480.000 qtGui.py:5283(highlightBlock) 303920.2050.0000.3670.000 qtGui.py:6294(match_keywords) 276000.1010.0000.1120.000 qtGui.py:6095(match_blanks) 268090.0180.0000.0180.000 qtGui.py:6672(trace_match) 245280.0500.0000.4030.000 qtGui.py:5973(colorRangeWithTag) 245260.1890.0000.3530.000 qtGui.py:6863(setTag) 166280.1150.0000.1430.000 qtGui.py:6776(setCurrentState) 166280.1080.0000.1720.000 qtGui.py:6757(getPrevState) 166280.0110.0000.0110.000 qtGui.py:6745(computeStateName) 53190.0360.0000.2550.000 qtGui.py:6239(match_eol_span) 53190.0230.0000.0630.000 qtGui.py:6654(skip_line) 52170.0280.0000.0770.000 qtGui.py:6449(match_seq) 51690.0320.0000.1300.000 qtGui.py:6501(match_span) 50300.0320.0000.1210.000 qtGui.py:6377(match_mark_previous) 28740.0090.0000.0110.000 qtGui.py:6539(match_span_helper) 12540.0230.0000.0870.000 qtGui.py:2980(add_command) 1177/11680.0450.0005.6940.005 qtGui.py:4760(eventFilter) This included a recoloring of a large block of text. Considering how many times recolor is called, it is doing very well. Perhaps the problem is not recolor, but some other inefficiency somewhere. Well, profiler will tell. BTW, did ensure that @nocolor is handled in a very fast fashion? It's critical to skip all of this machinery for huge log files/data I'll look into this. At present, not even @killcolor is working properly. Edward --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Feb 10, 2:43 pm, Edward K. Ream edream...@gmail.com wrote: Rev 1537 of the trunk contains the merge of the fast-color branch into Please report any problem immediately. Thanks. Hi. I have just tried new qt plug-in and found out that when I select any node that has some characters from unicode range, for the first time it is displayed correct, but when I select another node and then select again first node, unicode text is replaced by ?. I am using Windows Vista Leo Log Window... Leo 4.5.1 final, build 1.244 , September 14, 2008 python 2.5.1, qt version 263169 Windows 6, 0, 6001, 2, Service Pack 1 Rev 1537 Vitalije --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Tue, Feb 10, 2009 at 4:48 PM, Ville M. Vainio vivai...@gmail.com wrote: AttributeError: currentBlock This ubuntu has qt version 4.3.3-2ubuntu4.1 Would it be possible to disable syntax highlighter globally if qt doesn't have that method? I'd rather have leo *launchable*, even if we wouldn't have coloring. Actually, leo does launch and works. It just gets slow because the exceptions get dumped to stdout, and coloring does not work (obviously). -- Ville M. Vainio http://tinyurl.com/vainio --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
Actually this bug was introduced in revision 1447 if this information would be helpful. Vitalije. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
Rev 1537 of the trunk contains the merge of the fast-color branch into Please report any problem immediately. Thanks. Hi. I have just tried new qt plug-in and found out that when I select any node that has some characters from unicode range, for the first time it is displayed correct, but when I select another node and then select again first node, unicode text is replaced by ?. I am using Windows Vista Leo Log Window... Leo 4.5.1 final, build 1.244 , September 14, 2008 python 2.5.1, qt version 263169 Windows 6, 0, 6001, 2, Service Pack 1 Rev 1537 Actually this bug was introduced in revision 1447 if this information would be helpful. Vitalije. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Tue, Feb 10, 2009 at 3:43 PM, Edward K. Ream edream...@gmail.com wrote: Please report any problem immediately. Thanks. Try pasting e.g. qtGui.py file contents in a node. It hangs indefinitely. Apart from that, editing is very snappy now. -- Ville M. Vainio http://tinyurl.com/vainio --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Tue, Feb 10, 2009 at 12:18 PM, Ville M. Vainio vivai...@gmail.comwrote: Try pasting e.g. qtGui.py file contents in a node. It hangs indefinitely. Oops. I'll look into this soon. Edward --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Tue, Feb 10, 2009 at 8:18 PM, Ville M. Vainio vivai...@gmail.com wrote: On Tue, Feb 10, 2009 at 3:43 PM, Edward K. Ream edream...@gmail.com wrote: Please report any problem immediately. Thanks. Try pasting e.g. qtGui.py file contents in a node. It hangs indefinitely. And pretty much every time I press ctrl+c, I see this: File /home/ville/leo-editor/leo/plugins/qtGui.py, line 5284, in highlightBlock colorer.recolor(s) File /home/ville/leo-editor/leo/plugins/qtGui.py, line 6677, in recolor all_s = self.w.getAllText() File /home/ville/leo-editor/leo/plugins/qtGui.py, line 7943, in getAllText s = w.toPlainText() It seems it's not all that fast to do getAllText every time. Perhaps you should cache allText, and set it to None when text is edited. That way, during full recolor, you wouldn't be calling it all the time. -- Ville M. Vainio http://tinyurl.com/vainio --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Tue, Feb 10, 2009 at 12:29 PM, Ville M. Vainio vivai...@gmail.comwrote: It seems it's not all that fast to do getAllText every time. Good catch. For large text (as in a large paste) the reallocation of the entire string will surely stress the gc greatly. The big payoff is when calling rehighlight, so we can have rehighlight do the caching. We must be very careful to clear the cache when we reach the end of the complete highlighting. I'll look into this soon. Edward --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Tue, Feb 10, 2009 at 8:55 PM, Edward K. Ream edream...@gmail.com wrote: On Tue, Feb 10, 2009 at 12:29 PM, Ville M. Vainio vivai...@gmail.com wrote: It seems it's not all that fast to do getAllText every time. Good catch. For large text (as in a large paste) the reallocation of the entire string will surely stress the gc greatly. The big payoff is when calling rehighlight, so we can have rehighlight do the caching. We must be very careful to clear the cache when we reach the end of the complete highlighting. You can clear the cache on textChanged () signal. The text will never change behind your back if you follow that signal. Be sure to implement the cache so that all of leo should benefit from this, so that there is only one place in leo where it does toPlainText directly. Also remove toUnicode and use unicode(s) instead. -- Ville M. Vainio http://tinyurl.com/vainio --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Tue, Feb 10, 2009 at 12:59 PM, Ville M. Vainio vivai...@gmail.comwrote: You can clear the cache on textChanged () signal. The text will never change behind your back if you follow that signal. Be sure to implement the cache so that all of leo should benefit from this, so that there is only one place in leo where it does toPlainText directly. Also remove toUnicode and use unicode(s) instead. Excellent suggestions. I'll certainly user them. However, it looks like severe gc problems could still persist in the colorizing code. At present, initRecolor contains the following code: if restartString: [little snip] searchString = restartString + all_s[offset:] Haha. It looks like the hack of prepending the restart string to the text to be searched is going to be a problem. all_s[offset:] can be very large. True, in most cases restartString will almost always be empty, but a single worst case could crash Leo. That's not acceptable. This is why I love programming. The puzzles never end and they are much more interesting than sudoku :-) In this case, we see a fascinating interaction between gc issues and algorithmic issues. It looks like another strategic review is in order. Don't worry, there is no need for another giant post. The big difference now is that I know about QTextBlock. position. It's likely that reviving the old big state approach will allow initRecolor never to allocate any strings at all. In other words, QTextBlock.position should enable a proof that indices into preceding lines *are* valid. My plan is to revive the fast-color branch with the goal of calling w.getAllText exactly once per recolor, regardless of whether the recoloring is incremental or not. I expect to get something working in a day or so... Edward P. S. Alternatively, we might try another hack: scan backwards from all_s[offset] to find the nearest instance of restartString. Alas, this is not guaranteed to find the actual proper string. Consider, for example, a regex pattern rule: it may match several instances of the restartString, and without knowing what the regex is we can't know whether the regex pattern matcher will match what we want. EKR --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
Re: fast-color branch now merged with the trunk
On Wed, Feb 11, 2009 at 5:56 AM, Ville M. Vainio vivai...@gmail.com wrote: No hacks so actual thought should be needed. Just add caching and .. hacks OR actual thought... -- Ville M. Vainio http://tinyurl.com/vainio --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups leo-editor group. To post to this group, send email to leo-editor@googlegroups.com To unsubscribe from this group, send email to leo-editor+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/leo-editor?hl=en -~--~~~~--~~--~--~---
