[Zim-wiki] caching and progressively slower page loads
When Zim starts up, switching pages is nice and snappy.
https://www.youtube.com/watch?v=tl4_t6dW1l0
After a while, loading even the smallest page has a noticeable delay.
https://www.youtube.com/watch?v=euyGgEwLaIk
Note the time required to display the new page in addition to the time
before the cursor becomes active.
This is running a recent Github version (commit
e46a4b40a90d6903b6fa860bd72a4464dab7d704) with all add-ons disabled,
including built-in add-ons. However, it was originally reproduced on
older versions, so it is not a new effect.
A few backtraces with GDB during a page switch shows most of the time is
spent in this query from the "get_mytreeiter()" function in
zim/notebook/index/pages.py:
for i, row in enumerate(self.db.execute('''
SELECT * FROM pages WHERE parent=?
ORDER BY sortkey, name LIMIT 20 OFFSET ?
''',
(parent_id, offset)
)):
https://github.com/jaap-karssenberg/zim-desktop-wiki/blob/e46a4b40a90d6903b6fa860bd72a4464dab7d704/zim/notebook/index/pages.py#L804
This is being called from the main loop.
Traceback (most recent call first):
File
"/home/nathaniel/.local/lib/python2.7/site-packages/zim/notebook/index/pages.py",
line 805, in get_mytreeiter
(parent_id, offset)
File
"/home/nathaniel/.local/lib/python2.7/site-packages/zim/gui/pageindex.py",
line 224, in on_get_iter
return self.get_mytreeiter(treepath)
File
"/home/nathaniel/.local/lib/python2.7/site-packages/zim/gui/pageindex.py",
line 238, in on_iter_next
return self.on_get_iter(treepath)
File
"/home/nathaniel/.local/lib/python2.7/site-packages/zim/main/__init__.py",
line 703, in _run_main_loop
gtk.main()
File
"/home/nathaniel/.local/lib/python2.7/site-packages/zim/main/__init__.py",
line 656, in _run_cmd
self._run_main_loop(cmd)
File
"/home/nathaniel/.local/lib/python2.7/site-packages/zim/main/__init__.py",
line 626, in run
self._run_cmd(cmd, args) # test seam
File
"/home/nathaniel/.local/lib/python2.7/site-packages/zim/main/__init__.py",
line 845, in main
The query itself is fast, so it seems strange that so much time is spent
here. However, running Zim with "cprofile" for about a day shows there
are roughly 1.3e6 calls to the sqlite3 execute method, so even though
the time per call is less than a millisecond, the cumulative time is
considerable.
Wed Jan 17 10:33:24 2018zim.notebook.index.pages_1516168791.prof
43403639 function calls (42725089 primitive calls) in
86784.644 seconds
Ordered by: internal time
List reduced from 2168 to 4 due to restriction <4>
ncalls tottime percall cumtime percall filename:lineno(function)
1 85453.595 85453.595 86765.186 86765.186 {gtk._gtk.main}
1347825 1154.6850.001 1154.6860.001 {method 'execute' of
'sqlite3.Connection' objects}
5138231 50.8810.000 1207.4540.000
/home/nathaniel/.local/lib/python2.7/site-packages/zim/gui/pageindex.py:231(on_iter_next)
11 43.9533.996 48.9924.454
/home/nathaniel/.local/lib/python2.7/site-packages/zim/gui/widgets.py:3152(run)
By setting a breakpoint in GDB:
break _pysqlite_query_execute
and then using "ignore" and "info break", we can count how many queries
are run. In my notebook of about 1,400 pages, there are queries
amounting to roughly twice that on startup. When the pages begin loading
slowly, it runs correspondingly more queries.
Since "get_mytreeiter()" caches the page on each call, is it possible
that the cache is being duplicated or otherwise not working properly,
resulting in an excessive number of queries?
If the cache is working as intended, could it be changed so that it does
not become progressively slower?
Should I file a bug report on Github, or should I wait until there is
something more concrete to submit?
Nathaniel Beaver
P.S. Upon request, I am happy to provide full data and details on
collection methods.
Please let me know if there is something I overlooked or a flaw in the
methodology.
This is a follow-up to my message from December 29, 2017 with subject
"profiling zim for performance?"
___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help : https://help.launchpad.net/ListHelp
Re: [Zim-wiki] profiling zim for performance?
On 12/30/2017 03:40 PM, Stephen LARROQUE wrote:
This is most likely a memory sink issue if it becomes more sluggish over
time. Valgrind can be used to profile memory and garbage collection, or
since zim is a python program you can also use the memory_profiler module.
Thanks for the suggestion, but it's the CPU that spikes when I change
pages, not the memory.
But note that i never experienced such an issue (but i leave zim open
max for 1d,then i don't fear closing since it memorizes cursor position
and documents history). Maybe it's the fault of a plugin?
I suppose it's possible that the tray icon plugin is responsible, but I
don't think it's likely.
Le Sam. 30 Déc. 2017 05:53, Nathaniel Beaver <nathanielmbea...@gmail.com
<mailto:nathanielmbea...@gmail.com>> a écrit :
When I first launch Zim, it is quite snappy. However, after it's been
running a while, opening even the smallest pages becomes sluggish.
What would be a good method of profiling Zim's performance to help
understand this behavior?
I've tried using `strace', but it just shows Zim is polling a lot, and
I'm not sure how to interpret the results.
I am using Zim 0.65-4 on Debian Stretch and have also verified the same
slowdown on the latest Github version (0.67, SHA1 is
7a373a788194c36723bd0eed600291e513de9396).
I have not enabled any plugins besides the tray icon.
Nathaniel Beaver
___
Mailing list: https://launchpad.net/~zim-wiki
<https://launchpad.net/%7Ezim-wiki>
Post to : zim-wiki@lists.launchpad.net
<mailto:zim-wiki@lists.launchpad.net>
Unsubscribe : https://launchpad.net/~zim-wiki
<https://launchpad.net/%7Ezim-wiki>
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help : https://help.launchpad.net/ListHelp
[Zim-wiki] profiling zim for performance?
When I first launch Zim, it is quite snappy. However, after it's been running a while, opening even the smallest pages becomes sluggish. What would be a good method of profiling Zim's performance to help understand this behavior? I've tried using `strace', but it just shows Zim is polling a lot, and I'm not sure how to interpret the results. I am using Zim 0.65-4 on Debian Stretch and have also verified the same slowdown on the latest Github version (0.67, SHA1 is 7a373a788194c36723bd0eed600291e513de9396). I have not enabled any plugins besides the tray icon. Nathaniel Beaver ___ Mailing list: https://launchpad.net/~zim-wiki Post to : zim-wiki@lists.launchpad.net Unsubscribe : https://launchpad.net/~zim-wiki More help : https://help.launchpad.net/ListHelp
[Zim-wiki] jump window text alignment
Sometimes the paths in the jump window are too long, and the matched page name might not be visible. For example, a search for "power-manager" might return projects:software:bugs-and-feature-requests:debian-sections:xfce:xfce4 projects:software:bugs-and-feature-requests:debian-sections:xfce:xfce4 when the full paths are projects:software:bugs-and-feature-requests:debian-sections:xfce:xfce4-power-manager projects:software:bugs-and-feature-requests:debian-sections:xfce:xfce4-power-manager-plugins (See attached screen shots.) One option is just to make the jump window wider so that the full path is visible. However, this can be ungainly and is limited by screen size. Would it be possible to right-align the matched text in the jump windows, as in the attached screenshot mockup? A search for "power-manager" would return are:bugs-and-feature-requests:debian-sections:xfce:xfce4-power-manager -and-feature-requests:debian-sections:xfce:xfce4-power-manager-plugins This way, the matching text would always be visible. Alternately, Nathaniel Beaver ___ Mailing list: https://launchpad.net/~zim-wiki Post to : zim-wiki@lists.launchpad.net Unsubscribe : https://launchpad.net/~zim-wiki More help : https://help.launchpad.net/ListHelp
Re: [Zim-wiki] Too large notebook
On 09/19/2016 01:11 AM, Robert Zelník wrote: Hi all, I am a very happy Zim user, I use it for many years as my personal external memory. I mostly use flat page structure, that means most of the pages are in the same level of hierarchy and under the same folding/parent page. I also used to do this. Unfortunately, I found it does not play to Zim's strengths. What is the size in kilobytes of the largest pages in your notebook? This general notebook contains for about 3500 pages in the main level (with some subpages under some of these main pages) and it causes few (8-9) seconds of freeze after each new click on a page. I also have some much smaller notebooks and there the freeze doesn't occur. What should I do to get it faster? Would be helpful to divide the notebook in multiple sub-sections, or should I divide it in multiple notebooks? I would recommend dividing into multiple sub-sections, not dividing into multiple notebooks. I have good performance on a single notebook that totals around 13 MB, but almost all of the pages are less than 8 kilobytes, and no single page is more than 12 KB. There are slightly more than 1000 pages in about 400 directories. This might sound difficult to navigate, but because of the structured hierarchy I've built up and the "Jump to" command, it is quite manageable. Normally a deep hierarchy would be constraining, but because Zim updates internal links automatically, there is less cost to re-arranging pages. I use a hybrid of categorizing by year/date and categorizing by subject. Keeping things in separate categories also helps avoid unintentional duplication, another problem I experienced when trying to keep everything in a few large pages. What could cause the freeze? Is it some kind of indexing? The task list plugin does index every file, but even that only takes a few seconds to start up for me, and usually has a latency less than a second when updating. I think the main issue you are experiencing is that Zim is both a text editor and a markup language renderer, and this does not mix well with large single pages. Since Zim pages are just text files, you could use a text editor that does not try to render the pages, but that would remove most of the benefits of Zim. However, with some care, you can use a different text editor when splitting up the larger pages. For example, you could create several "stub" pages with Zim, then close Zim and use your favorite text editor to cut and paste sections of the larger pages into the smaller stub pages. Just make sure you don't move any internal links or embedded images during this process, because Zim won't be able to fix them automatically. Also, I would recommend making a full backup of your notebook before undertaking this process. -- Robert Zelník http://about.me/robert.zelnik ___ Mailing list: https://launchpad.net/~zim-wiki Post to : zim-wiki@lists.launchpad.net Unsubscribe : https://launchpad.net/~zim-wiki More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~zim-wiki Post to : zim-wiki@lists.launchpad.net Unsubscribe : https://launchpad.net/~zim-wiki More help : https://help.launchpad.net/ListHelp
[Zim-wiki] path link checking plugin
I've written a plugin to check that the target of a file link exists
(see prototype attached).
However, I'd like to turn the broken links a different color by
accessing the gtk.TextTag corresponding to the link and changing the
style with something like
tag.set_property('foreground', 'red')
Unfortunately, I am having some difficulty finding the actual TextTags
in the parse tree. Right now I am using tree.findall(zim.formats.LINK)
to get all the links, but the node.tag property is just a string, not a
TextTag.
Any suggestions?
Nathaniel Beaver
P.S. Here is the GitHub page if it helps:
https://github.com/nbeaver/zim-pathchecker-plugin
and here is the relevant bug report:
https://bugs.launchpad.net/zim/+bug/1419531
#!/usr/bin/env python
# -*- coding: utf-8 -*-
'''Path check plugin based on zeitgeist-logger plugin.'''
import sys
import os.path
import urlparse
from zim.plugins import PluginClass, ObjectExtension, extends
from zim.signals import SIGNAL_AFTER
import zim.formats
import zim.parsing
import logging
import inspect
logger = logging.getLogger('zim.plugins.pathcheck')
class PathChecker(PluginClass):
plugin_info = {
'name': "Path Checker", # T: plugin name
'description': "Checks for broken paths in links.", # T: plugin description
'author': "Nathaniel Beaver",
}
@extends('PageView')
class PageViewExtension(ObjectExtension):
def __init__(self, plugin, pageview):
self.plugin = plugin
self.connectto(obj=pageview.ui, signal='open-page', handler=self.on_open_page, order=SIGNAL_AFTER)
def yield_link_nodes(self, page):
tree = page.get_parsetree()
if tree:
for node in tree.findall(zim.formats.LINK):
href = node.attrib.pop('href')
link_type = zim.parsing.link_type(href)
yield link_type, href, node
def on_open_page(self, ui, page, path):
logger.debug("Opened page: %s", page.name)
for link_type, href, node in self.yield_link_nodes(page):
if link_type == 'file':
if href.startswith('file://'):
path = urlparse.urlparse(href).path
else:
path = os.path.expanduser(href)
if not os.path.exists(path):
logger.debug("Broken path: %s", path)
___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help : https://help.launchpad.net/ListHelp
