Re: [Zim-wiki] Making indexing optional ?

[Jaap Karssenberg]

Hi Mario,

I did some testing myself last night, will write up the full log in the dev
wiki later but here is the summary:

Generated 3 random notebooks:
1/ 16_000+ files over 600+ folders, total size on disk 200Mb (many small
files)
2/ 16_000+ files over 600+ folders, total size on disk 3Gb (many larger
files)
3/ the same as #1 but with a single ".txt" attachment of 1Gb

Indexing with "zim --index" takes 3-4 minutes for #1 & #3, but over 60
minutes (!) for #2

Once the indexing is complete I open the notebook with zim. For all 3
cases, the UI responds within seconds. The background index check runs for
1-2 minutes but does not interfere with using the application already.

Conclusions:
- Large text attachments are handled (ignored) correctly by the indexer
- Indexing cache allows fast opening of a very large notebook when the
index is up to date
- Indexing speed scales with the amount of text that needs to be read and
parsed. Parsing is not super fast and might be improved, but I consider the
3Gb notebook an extreme scenario; you can spent a life time typing in 3Gb
of text notes ;)

Further thoughts:
- When testing a new version (or e.g. switching between master and develop
branches) you may need to run an index update first for fare comparison,
sometimes an update is needed when the database scheme changed
- The index uses the "mtime" time stamps on files and folders. If your
notebook needs index updates every time you open it, and you have a
non-standard file system, it might be the file system does not represent
mtimes correctly.
- By default the index is in the user cache folder (~/.cache on unix) so
using the same notebook from multiple systems may result in indexing the
same changes multiple times. You can choose to store the index in the
notebook itself (see manual for the "shared" notebook property), however
this may cause trouble if you try to access from both systems at the same
time.

Hope this helps to clarify what to expect,

Jaap


On Tue, Jun 29, 2021 at 12:02 PM Mario Bezzi <
subscriptions.mario.be...@gmail.com> wrote:

> Hi Jaap, thank you for all your help on this.
>
> I just tested the development branch which seems to include your mentioned
> fix. Sadly zim startup took approximately the same time as before.
> Maybe my use case is abnormal? If so I don't want you to spend lots of
> time on this.
>
> Below the output of ./zim.py -V -D. All the time is spent in
> BackgroundCheck. Is this what you would expect? May we (I?) add debug
> messages to track the time spent indexing large files?
>
> Thank you,
> mario
>
> -
>
> mario_bezzi@XPS-15-9560:~/Downloads/zim-desktop-wiki-develop$ ./zim.py -V
> -D
> DEBUG: Loading config from:  /home/mario_bezzi/.config/zim/preferences.conf>
> DEBUG: New extendable:  0x7f99b7a5eaf0>
> DEBUG: Loading plugin: pageindex
> DEBUG: Loading plugin: pathbar
> DEBUG: Loading plugin: insertsymbol
> DEBUG: Loading plugin: printtobrowser
> DEBUG: Loading plugin: versioncontrol
> DEBUG: Loading plugin: attachmentbrowser
> DEBUG: Loading plugin: bookmarksbar
> DEBUG: Loading plugin: journal
> INFO: This is zim 0.73.5
> DEBUG: Python version is sys.version_info(major=3, minor=8, micro=5,
> releaselevel='final', serial=0)
> DEBUG: Platform is posix
> DEBUG: Running from a source dir:
> /home/mario_bezzi/Downloads/zim-desktop-wiki-develop
> DEBUG: Set XDG_DATA_HOME to /home/mario_bezzi/.local/share
> DEBUG: Set XDG_DATA_DIRS to [,  /usr/local/share>, , ]
> DEBUG: Set XDG_CONFIG_HOME to /home/mario_bezzi/.config
> DEBUG: Set XDG_CONFIG_DIRS to [, ]
> DEBUG: Set XDG_CACHE_HOME to /home/mario_bezzi/.cache
> DEBUG: Connecting to /run/user/1000/zim-0.73.5-43e7ec27
> DEBUG: Got error in dispatch: No such file or directory
> DEBUG: Starting primary process
> DEBUG: Start listening on: /run/user/1000/zim-0.73.5-43e7ec27
> DEBUG: Loading config from:  0x7f99b5719040>
> DEBUG: Loading config from:
> /home/mario_bezzi/Dropbox/Documents/Wikis/notebook.zim
> DEBUG: Loading config from:
> /home/mario_bezzi/Dropbox/Documents/Wikis/notebook.zim
> DEBUG: Loading config from:
> /home/mario_bezzi/.cache/zim/notebook-home_mario_bezzi_Dropbox_Documents_Wikis/state.conf
> DEBUG: New extendable: 
> DEBUG: Load extension:  'zim.plugins.journal.JournalNotebookExtension'>
> DEBUG: Load extension:  'zim.plugins.versioncontrol.VersionControlNotebookExtension'>
> INFO: No VCS detected
> DEBUG: Loading config from:  /home/mario_bezzi/.config/zim/style.conf>
> DEBUG: Autosave interval: 15 - use threads: True
> DEBUG: Loading config from:  /home/mario_bezzi/.config/zim/customtools/command shell-usercreated.desktop>
> INFO: Page changed on disk: Home:ZABEXPRF:ZX Manuals
> INFO: Open page: Home:ZABEXPRF:ZX Manuals (Home:ZABEXPRF:ZX Manuals)
> DEBUG: New extendable:  0x7f99b56bee00 (zim+gui+notebookview+NotebookView at 0x160)>
> DEBUG: Load extension:  'zim.plugins.attachmentbrowser.AttachmentBrowserWindowExtension'>
> DEBUG: Action: toggle_panes(False)
> DEBUG: Action: 

Re: [Zim-wiki] Making indexing optional ?

[Mario Bezzi]

Hi Jaap, thank you for all your help on this.

I just tested the development branch which seems to include your 
mentioned fix. Sadly zim startup took approximately the same time as 
before.
Maybe my use case is abnormal? If so I don't want you to spend lots of 
time on this.


Below the output of ./zim.py -V -D. All the time is spent in 
BackgroundCheck. Is this what you would expect? May we (I?) add debug 
messages to track the time spent indexing large files?


Thank you,
mario

-

mario_bezzi@XPS-15-9560:~/Downloads/zim-desktop-wiki-develop$ ./zim.py -V -D
DEBUG: Loading config from: /home/mario_bezzi/.config/zim/preferences.conf>
DEBUG: New extendable: 0x7f99b7a5eaf0>

DEBUG: Loading plugin: pageindex
DEBUG: Loading plugin: pathbar
DEBUG: Loading plugin: insertsymbol
DEBUG: Loading plugin: printtobrowser
DEBUG: Loading plugin: versioncontrol
DEBUG: Loading plugin: attachmentbrowser
DEBUG: Loading plugin: bookmarksbar
DEBUG: Loading plugin: journal
INFO: This is zim 0.73.5
DEBUG: Python version is sys.version_info(major=3, minor=8, micro=5, 
releaselevel='final', serial=0)

DEBUG: Platform is posix
DEBUG: Running from a source dir: 
/home/mario_bezzi/Downloads/zim-desktop-wiki-develop

DEBUG: Set XDG_DATA_HOME to /home/mario_bezzi/.local/share
DEBUG: Set XDG_DATA_DIRS to [, /usr/local/share>, , ]

DEBUG: Set XDG_CONFIG_HOME to /home/mario_bezzi/.config
DEBUG: Set XDG_CONFIG_DIRS to [, ]
DEBUG: Set XDG_CACHE_HOME to /home/mario_bezzi/.cache
DEBUG: Connecting to /run/user/1000/zim-0.73.5-43e7ec27
DEBUG: Got error in dispatch: No such file or directory
DEBUG: Starting primary process
DEBUG: Start listening on: /run/user/1000/zim-0.73.5-43e7ec27
DEBUG: Loading config from: 0x7f99b5719040>
DEBUG: Loading config from: 
/home/mario_bezzi/Dropbox/Documents/Wikis/notebook.zim
DEBUG: Loading config from: 
/home/mario_bezzi/Dropbox/Documents/Wikis/notebook.zim
DEBUG: Loading config from: 
/home/mario_bezzi/.cache/zim/notebook-home_mario_bezzi_Dropbox_Documents_Wikis/state.conf

DEBUG: New extendable: 
DEBUG: Load extension: 'zim.plugins.journal.JournalNotebookExtension'>
DEBUG: Load extension: 'zim.plugins.versioncontrol.VersionControlNotebookExtension'>

INFO: No VCS detected
DEBUG: Loading config from: /home/mario_bezzi/.config/zim/style.conf>

DEBUG: Autosave interval: 15 - use threads: True
DEBUG: Loading config from: /home/mario_bezzi/.config/zim/customtools/command shell-usercreated.desktop>

INFO: Page changed on disk: Home:ZABEXPRF:ZX Manuals
INFO: Open page: Home:ZABEXPRF:ZX Manuals (Home:ZABEXPRF:ZX Manuals)
DEBUG: New extendable: 0x7f99b56bee00 (zim+gui+notebookview+NotebookView at 0x160)>
DEBUG: Load extension: 'zim.plugins.attachmentbrowser.AttachmentBrowserWindowExtension'>

DEBUG: Action: toggle_panes(False)
DEBUG: Action: toggle_panes(True)
DEBUG: Load extension: 'zim.plugins.insertsymbol.InsertSymbolPageViewExtension'>
DEBUG: Load extension: 'zim.plugins.journal.JournalNotebookViewExtension'>
DEBUG: Load extension: 'zim.plugins.pageindex.PageIndexNotebookViewExtension'>
DEBUG: Load extension: 'zim.plugins.printtobrowser.PrintToBrowserPageViewExtension'>
DEBUG: New extendable: (zim+gui+mainwindow+MainWindow at 0x1b9a270)>
DEBUG: Load extension: 'zim.plugins.bookmarksbar.BookmarksBarMainWindowExtension'>
DEBUG: Load extension: 'zim.plugins.pathbar.PathBarMainWindowExtension'>
DEBUG: Load extension: 'zim.plugins.versioncontrol.VersionControlMainWindowExtension'>

DEBUG: Accelmap: /home/mario_bezzi/.config/zim/accelmap
DEBUG: Add window: MainWindow
DEBUG: BackgroundCheck started
DEBUG: BackgroundCheck finished



On 6/26/21 7:55 PM, Jaap Karssenberg wrote:

Hi Mario,

Just pushed a fix (29bdea) that improves how we check whether files 
are a zim page or not. Now max 50 characters are being read at the 
start of the file when indexing. If your large files are not "line 
based" (thus resulting in a very long read when trying to read the 
first line) this should fix the issue.


Regards,

Jaap


On Sat, Apr 24, 2021 at 10:06 AM Mario Bezzi 
> wrote:


Hi Jaap, thank you for your help on this.

To give you some more details: Of the 3000+ files which size sums
up to 2GB, the top 500 account for 1.6GB. Among these the average
size is 3.5MB, and each of the top three is in the 250MB range.

Please let me know if there is anything I can do to help testing
your fix,
mario

On 4/23/21 2:55 PM, Jaap Karssenberg wrote:

Hi Mario,

That is not the result I hoped for :(   I will need to generate
some random large text files to test & debug on my end.

Regards,

Jaap


On Fri, Apr 23, 2021 at 12:59 PM Mario Bezzi
mailto:subscriptions.mario.be...@gmail.com>> wrote:

I think I submitted my request circa 2014 under the previous
bug tracking system - was it hosted by Ubuntu-one? - but yes,
the idea is similar.

I just downloaded the development version, 

Re: [Zim-wiki] Making indexing optional ?

[Jaap Karssenberg]

Hi Mario,

Just pushed a fix (29bdea) that improves how we check whether files are a
zim page or not. Now max 50 characters are being read at the start of the
file when indexing. If your large files are not "line based" (thus
resulting in a very long read when trying to read the first line) this
should fix the issue.

Regards,

Jaap


On Sat, Apr 24, 2021 at 10:06 AM Mario Bezzi <
subscriptions.mario.be...@gmail.com> wrote:

> Hi Jaap, thank you for your help on this.
>
> To give you some more details: Of the 3000+ files which size sums up to
> 2GB, the top 500 account for 1.6GB. Among these the average size is 3.5MB,
> and each of the top three is in the 250MB range.
>
> Please let me know if there is anything I can do to help testing your fix,
> mario
>
> On 4/23/21 2:55 PM, Jaap Karssenberg wrote:
>
> Hi Mario,
>
> That is not the result I hoped for :(   I will need to generate some
> random large text files to test & debug on my end.
>
> Regards,
>
> Jaap
>
>
> On Fri, Apr 23, 2021 at 12:59 PM Mario Bezzi <
> subscriptions.mario.be...@gmail.com> wrote:
>
>> I think I submitted my request circa 2014 under the previous bug tracking
>> system - was it hosted by Ubuntu-one? - but yes, the idea is similar.
>>
>> I just downloaded the development version, extracted it into a temporary
>> folder, and ran it via the ./zim.py command.
>>
>> Indexing took some 15 minutes. Below a snapshot of what top was saying
>> about the execution.
>>
>> top - 12:45:28 up 3 days, 16:12,  1 user,  load average: 1.87, 1.92, 2.48
>> Tasks: 356 total,   3 running, 353 sleeping,   0 stopped,   0 zombie
>> %Cpu(s): 13.0 us,  5.4 sy,  0.0 ni, 81.6 id,  0.0 wa,  0.0 hi,  0.0 si,
>> 0.0 st
>> MiB Mem :  31658.1 total,320.9 free,  19312.0 used,  12025.3
>> buff/cache
>> MiB Swap:976.0 total,  0.0 free,976.0 used.  10085.6 avail
>> Mem
>>
>> PID USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+
>> COMMAND
>>
>>  159310 mario_b+  20   0  771220  80184  43420 R 100.0   0.2  *14:42.13
>> zim.py*
>>
>> Please let me know if there is more I can do.
>>
>> Thank you,
>> mario
>>
>> On 4/23/21 11:25 AM, Jaap Karssenberg wrote:
>>
>> Yes that explains, those large files will have a big impact on the
>> indexer.
>>
>> You are referring to this issue: Make indexer ignore text files that are
>> not zim pages · Issue #907 · zim-desktop-wiki/zim-desktop-wiki (github.com)
>>  which
>> is fixed in the development branch and will be in the next release.
>>
>> With that fix the indexer will read the first line of each file to decide
>> whether it is a zim file or not, and if not it will not try to access the
>> contents.
>>
>> Would be great if you have a chance to test the development branch and
>> see whether it works in practice for your case !
>>
>> -- Jaap
>>
>>
>> On Thu, Apr 22, 2021 at 7:32 PM Mario Bezzi <
>> subscriptions.mario.be...@gmail.com> wrote:
>>
>>> The folder contains 3118 ".txt" files, for a total of 2GB of data. Some
>>> large txt files are attachments. A long time ago I submitted a request to
>>> avoid indexing these. Not sure it has been fulfilled though.
>>>
>>> Thank you,
>>> mario
>>>
>>> On 4/8/21 7:32 PM, Jaap Karssenberg wrote:
>>>
>>> Can you indicate how big your notebook folder is? Either an extreme
>>> case, or some bug making it take much longer than needed.
>>>
>>> Op do 8 apr. 2021 15:59 schreef Mario Bezzi <
>>> subscriptions.mario.be...@gmail.com>:
>>>
 Thanks Jaap, I was not aware of this.

 To give you an idea, I just restarted Zim, and indexing kept a
 processor 100% busy for 13 minutes to come to an end.  It was nice if this
 could be avoided.

 Thank you,
 mario

 On 4/8/21 10:06 AM, Jaap Karssenberg wrote:

 The indexing is not used for searching alone, it is also needed to e.g.
 present the page tree in the side pane and to track links

 Op do 8 apr. 2021 09:34 schreef Mario Bezzi <
 subscriptions.mario.be...@gmail.com>:

> Hello,
>
> I may be the only one, but with my quite large notebooks I do find the
> search function impractical, and for this reason I never use it.
> Still,
> when it starts, Zim goes crazy for a long time indexing, and I came to
> the conclusion that this is normal.
>
> If this is the case, I would like to file a requirement to add the
> ability to make indexing optional.
>
> Thank you,
> mario
>
> ___
> Mailing list: https://launchpad.net/~zim-wiki
> Post to : zim-wiki@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~zim-wiki
> More help   : https://help.launchpad.net/ListHelp
>


>>>
>>
>
___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help  

Re: [Zim-wiki] Making indexing optional ?

[Mario Bezzi]

Hi Jaap, thank you for your help on this.

To give you some more details: Of the 3000+ files which size sums up to 
2GB, the top 500 account for 1.6GB. Among these the average size is 
3.5MB, and each of the top three is in the 250MB range.


Please let me know if there is anything I can do to help testing your fix,
mario

On 4/23/21 2:55 PM, Jaap Karssenberg wrote:

Hi Mario,

That is not the result I hoped for :(   I will need to generate some 
random large text files to test & debug on my end.


Regards,

Jaap


On Fri, Apr 23, 2021 at 12:59 PM Mario Bezzi 
> wrote:


I think I submitted my request circa 2014 under the previous bug
tracking system - was it hosted by Ubuntu-one? - but yes, the idea
is similar.

I just downloaded the development version, extracted it into a
temporary folder, and ran it via the ./zim.py command.

Indexing took some 15 minutes. Below a snapshot of what top was
saying about the execution.

top - 12:45:28 up 3 days, 16:12,  1 user,  load average: 1.87,
1.92, 2.48
Tasks: 356 total,   3 running, 353 sleeping,   0 stopped,   0 zombie
%Cpu(s): 13.0 us,  5.4 sy,  0.0 ni, 81.6 id,  0.0 wa, 0.0 hi,  0.0
si,  0.0 st
MiB Mem :  31658.1 total,    320.9 free,  19312.0 used, 12025.3
buff/cache
MiB Swap:    976.0 total,  0.0 free,    976.0 used. 10085.6
avail Mem

    PID USER  PR  NI    VIRT    RES    SHR S  %CPU %MEM
TIME+ COMMAND
159310 mario_b+  20   0  771220 80184  43420 R 100.0   0.2
*14:42.13 zim.py*

Please let me know if there is more I can do.

Thank you,
mario

On 4/23/21 11:25 AM, Jaap Karssenberg wrote:

Yes that explains, those large files will have a big impact on
the indexer.

You are referring to this issue: Make indexer ignore text files
that are not zim pages · Issue #907 ·
zim-desktop-wiki/zim-desktop-wiki (github.com)
 which
is fixed in the development branch and will be in the next release.

With that fix the indexer will read the first line of each file
to decide whether it is a zim file or not, and if not it will not
try to access the contents.

Would be great if you have a chance to test the development
branch and see whether it works in practice for your case !

-- Jaap


On Thu, Apr 22, 2021 at 7:32 PM Mario Bezzi
mailto:subscriptions.mario.be...@gmail.com>> wrote:

The folder contains 3118 ".txt" files, for a total of 2GB of
data. Some large txt files are attachments. A long time ago I
submitted a request to avoid indexing these. Not sure it has
been fulfilled though.

Thank you,
mario

On 4/8/21 7:32 PM, Jaap Karssenberg wrote:

Can you indicate how big your notebook folder is? Either an
extreme case, or some bug making it take much longer than
needed.

Op do 8 apr. 2021 15:59 schreef Mario Bezzi
mailto:subscriptions.mario.be...@gmail.com>>:

Thanks Jaap, I was not aware of this.

To give you an idea, I just restarted Zim, and indexing
kept a processor 100% busy for 13 minutes to come to an
end.  It was nice if this could be avoided.

Thank you,
mario

On 4/8/21 10:06 AM, Jaap Karssenberg wrote:

The indexing is not used for searching alone, it is
also needed to e.g. present the page tree in the side
pane and to track links

Op do 8 apr. 2021 09:34 schreef Mario Bezzi
mailto:subscriptions.mario.be...@gmail.com>>:

Hello,

I may be the only one, but with my quite large
notebooks I do find the
search function impractical, and for this reason I
never use it. Still,
when it starts, Zim goes crazy for a long time
indexing, and I came to
the conclusion that this is normal.

If this is the case, I would like to file a
requirement to add the
ability to make indexing optional.

Thank you,
mario

___
Mailing list: https://launchpad.net/~zim-wiki

Post to     : zim-wiki@lists.launchpad.net

Unsubscribe : https://launchpad.net/~zim-wiki

More help   : https://help.launchpad.net/ListHelp










___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net

Re: [Zim-wiki] Making indexing optional ?

[Jaap Karssenberg]

Hi Mario,

That is not the result I hoped for :(   I will need to generate some random
large text files to test & debug on my end.

Regards,

Jaap


On Fri, Apr 23, 2021 at 12:59 PM Mario Bezzi <
subscriptions.mario.be...@gmail.com> wrote:

> I think I submitted my request circa 2014 under the previous bug tracking
> system - was it hosted by Ubuntu-one? - but yes, the idea is similar.
>
> I just downloaded the development version, extracted it into a temporary
> folder, and ran it via the ./zim.py command.
>
> Indexing took some 15 minutes. Below a snapshot of what top was saying
> about the execution.
>
> top - 12:45:28 up 3 days, 16:12,  1 user,  load average: 1.87, 1.92, 2.48
> Tasks: 356 total,   3 running, 353 sleeping,   0 stopped,   0 zombie
> %Cpu(s): 13.0 us,  5.4 sy,  0.0 ni, 81.6 id,  0.0 wa,  0.0 hi,  0.0 si,
> 0.0 st
> MiB Mem :  31658.1 total,320.9 free,  19312.0 used,  12025.3 buff/cache
> MiB Swap:976.0 total,  0.0 free,976.0 used.  10085.6 avail Mem
>
> PID USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+
> COMMAND
>
>  159310 mario_b+  20   0  771220  80184  43420 R 100.0   0.2  *14:42.13
> zim.py*
>
> Please let me know if there is more I can do.
>
> Thank you,
> mario
>
> On 4/23/21 11:25 AM, Jaap Karssenberg wrote:
>
> Yes that explains, those large files will have a big impact on the
> indexer.
>
> You are referring to this issue: Make indexer ignore text files that are
> not zim pages · Issue #907 · zim-desktop-wiki/zim-desktop-wiki (github.com)
>  which
> is fixed in the development branch and will be in the next release.
>
> With that fix the indexer will read the first line of each file to decide
> whether it is a zim file or not, and if not it will not try to access the
> contents.
>
> Would be great if you have a chance to test the development branch and see
> whether it works in practice for your case !
>
> -- Jaap
>
>
> On Thu, Apr 22, 2021 at 7:32 PM Mario Bezzi <
> subscriptions.mario.be...@gmail.com> wrote:
>
>> The folder contains 3118 ".txt" files, for a total of 2GB of data. Some
>> large txt files are attachments. A long time ago I submitted a request to
>> avoid indexing these. Not sure it has been fulfilled though.
>>
>> Thank you,
>> mario
>>
>> On 4/8/21 7:32 PM, Jaap Karssenberg wrote:
>>
>> Can you indicate how big your notebook folder is? Either an extreme case,
>> or some bug making it take much longer than needed.
>>
>> Op do 8 apr. 2021 15:59 schreef Mario Bezzi <
>> subscriptions.mario.be...@gmail.com>:
>>
>>> Thanks Jaap, I was not aware of this.
>>>
>>> To give you an idea, I just restarted Zim, and indexing kept a processor
>>> 100% busy for 13 minutes to come to an end.  It was nice if this could be
>>> avoided.
>>>
>>> Thank you,
>>> mario
>>>
>>> On 4/8/21 10:06 AM, Jaap Karssenberg wrote:
>>>
>>> The indexing is not used for searching alone, it is also needed to e.g.
>>> present the page tree in the side pane and to track links
>>>
>>> Op do 8 apr. 2021 09:34 schreef Mario Bezzi <
>>> subscriptions.mario.be...@gmail.com>:
>>>
 Hello,

 I may be the only one, but with my quite large notebooks I do find the
 search function impractical, and for this reason I never use it. Still,
 when it starts, Zim goes crazy for a long time indexing, and I came to
 the conclusion that this is normal.

 If this is the case, I would like to file a requirement to add the
 ability to make indexing optional.

 Thank you,
 mario

 ___
 Mailing list: https://launchpad.net/~zim-wiki
 Post to : zim-wiki@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~zim-wiki
 More help   : https://help.launchpad.net/ListHelp

>>>
>>>
>>
>
___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help   : https://help.launchpad.net/ListHelp

Re: [Zim-wiki] Making indexing optional ?

[Mario Bezzi]

I think I submitted my request circa 2014 under the previous bug 
tracking system - was it hosted by Ubuntu-one? - but yes, the idea is 
similar.


I just downloaded the development version, extracted it into a temporary 
folder, and ran it via the ./zim.py command.


Indexing took some 15 minutes. Below a snapshot of what top was saying 
about the execution.


top - 12:45:28 up 3 days, 16:12,  1 user,  load average: 1.87, 1.92, 2.48
Tasks: 356 total,   3 running, 353 sleeping,   0 stopped,   0 zombie
%Cpu(s): 13.0 us,  5.4 sy,  0.0 ni, 81.6 id,  0.0 wa,  0.0 hi, 0.0 si,  
0.0 st

MiB Mem :  31658.1 total,    320.9 free,  19312.0 used,  12025.3 buff/cache
MiB Swap:    976.0 total,  0.0 free,    976.0 used.  10085.6 avail Mem

    PID USER  PR  NI    VIRT    RES    SHR S  %CPU  %MEM TIME+ COMMAND
159310 mario_b+  20   0  771220  80184  43420 R 100.0   0.2 *14:42.13 
zim.py*


Please let me know if there is more I can do.

Thank you,
mario

On 4/23/21 11:25 AM, Jaap Karssenberg wrote:
Yes that explains, those large files will have a big impact on the 
indexer.


You are referring to this issue: Make indexer ignore text files that 
are not zim pages · Issue #907 · zim-desktop-wiki/zim-desktop-wiki 
(github.com) 
 which 
is fixed in the development branch and will be in the next release.


With that fix the indexer will read the first line of each file to 
decide whether it is a zim file or not, and if not it will not try to 
access the contents.


Would be great if you have a chance to test the development branch and 
see whether it works in practice for your case !


-- Jaap


On Thu, Apr 22, 2021 at 7:32 PM Mario Bezzi 
> wrote:


The folder contains 3118 ".txt" files, for a total of 2GB of data.
Some large txt files are attachments. A long time ago I submitted
a request to avoid indexing these. Not sure it has been fulfilled
though.

Thank you,
mario

On 4/8/21 7:32 PM, Jaap Karssenberg wrote:

Can you indicate how big your notebook folder is? Either an
extreme case, or some bug making it take much longer than needed.

Op do 8 apr. 2021 15:59 schreef Mario Bezzi
mailto:subscriptions.mario.be...@gmail.com>>:

Thanks Jaap, I was not aware of this.

To give you an idea, I just restarted Zim, and indexing kept
a processor 100% busy for 13 minutes to come to an end.  It
was nice if this could be avoided.

Thank you,
mario

On 4/8/21 10:06 AM, Jaap Karssenberg wrote:

The indexing is not used for searching alone, it is also
needed to e.g. present the page tree in the side pane and to
track links

Op do 8 apr. 2021 09:34 schreef Mario Bezzi
mailto:subscriptions.mario.be...@gmail.com>>:

Hello,

I may be the only one, but with my quite large notebooks
I do find the
search function impractical, and for this reason I never
use it. Still,
when it starts, Zim goes crazy for a long time indexing,
and I came to
the conclusion that this is normal.

If this is the case, I would like to file a requirement
to add the
ability to make indexing optional.

Thank you,
mario

___
Mailing list: https://launchpad.net/~zim-wiki

Post to     : zim-wiki@lists.launchpad.net

Unsubscribe : https://launchpad.net/~zim-wiki

More help   : https://help.launchpad.net/ListHelp








___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help   : https://help.launchpad.net/ListHelp

Re: [Zim-wiki] Making indexing optional ?

[Jaap Karssenberg]

Yes that explains, those large files will have a big impact on the indexer.

You are referring to this issue: Make indexer ignore text files that are
not zim pages · Issue #907 · zim-desktop-wiki/zim-desktop-wiki (github.com)
 which is
fixed in the development branch and will be in the next release.

With that fix the indexer will read the first line of each file to decide
whether it is a zim file or not, and if not it will not try to access the
contents.

Would be great if you have a chance to test the development branch and see
whether it works in practice for your case !

-- Jaap


On Thu, Apr 22, 2021 at 7:32 PM Mario Bezzi <
subscriptions.mario.be...@gmail.com> wrote:

> The folder contains 3118 ".txt" files, for a total of 2GB of data. Some
> large txt files are attachments. A long time ago I submitted a request to
> avoid indexing these. Not sure it has been fulfilled though.
>
> Thank you,
> mario
>
> On 4/8/21 7:32 PM, Jaap Karssenberg wrote:
>
> Can you indicate how big your notebook folder is? Either an extreme case,
> or some bug making it take much longer than needed.
>
> Op do 8 apr. 2021 15:59 schreef Mario Bezzi <
> subscriptions.mario.be...@gmail.com>:
>
>> Thanks Jaap, I was not aware of this.
>>
>> To give you an idea, I just restarted Zim, and indexing kept a processor
>> 100% busy for 13 minutes to come to an end.  It was nice if this could be
>> avoided.
>>
>> Thank you,
>> mario
>>
>> On 4/8/21 10:06 AM, Jaap Karssenberg wrote:
>>
>> The indexing is not used for searching alone, it is also needed to e.g.
>> present the page tree in the side pane and to track links
>>
>> Op do 8 apr. 2021 09:34 schreef Mario Bezzi <
>> subscriptions.mario.be...@gmail.com>:
>>
>>> Hello,
>>>
>>> I may be the only one, but with my quite large notebooks I do find the
>>> search function impractical, and for this reason I never use it. Still,
>>> when it starts, Zim goes crazy for a long time indexing, and I came to
>>> the conclusion that this is normal.
>>>
>>> If this is the case, I would like to file a requirement to add the
>>> ability to make indexing optional.
>>>
>>> Thank you,
>>> mario
>>>
>>> ___
>>> Mailing list: https://launchpad.net/~zim-wiki
>>> Post to : zim-wiki@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~zim-wiki
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>
>>
>
___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help   : https://help.launchpad.net/ListHelp

Re: [Zim-wiki] Making indexing optional ?

[John Edge]

Comment from a relatively recent user of Zim-Wiki with relatively small wikis 
so far, running on Windows. A couple of times I have noticed that the Zim 
process was taking 10 to 15% of the CPU for several minutes. When I killed the 
process, and restarted Zim, this high usage level did not return. 

tksJohn Edge jr_e...@yahoo.com 

On Thursday, 8 April 2021, 16:00:00 CEST, Mario Bezzi 
 wrote:  
 
  Thanks Jaap, I was not aware of this.
 
 To give you an idea, I just restarted Zim, and indexing kept a processor 100% 
busy for 13 minutes to come to an end.  It was nice if this could be avoided.
 
 Thank you,
 mario
 
 On 4/8/21 10:06 AM, Jaap Karssenberg wrote:
  
 
The indexing is not used for searching alone, it is also needed to e.g. present 
the page tree in the side pane and to track links 
  Op do 8 apr. 2021 09:34 schreef Mario Bezzi 
:
  
Hello,
 
 I may be the only one, but with my quite large notebooks I do find the 
 search function impractical, and for this reason I never use it. Still, 
 when it starts, Zim goes crazy for a long time indexing, and I came to 
 the conclusion that this is normal.
 
 If this is the case, I would like to file a requirement to add the 
 ability to make indexing optional.
 
 Thank you,
 mario
 
 ___
 Mailing list: https://launchpad.net/~zim-wiki
 Post to     : zim-wiki@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~zim-wiki
 More help   : https://help.launchpad.net/ListHelp
 
  
 ___
Mailing list: https://launchpad.net/~zim-wiki
Post to    : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help  : https://help.launchpad.net/ListHelp
  ___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help   : https://help.launchpad.net/ListHelp

Re: [Zim-wiki] Making indexing optional ?

[Jaap Karssenberg]

Can you indicate how big your notebook folder is? Either an extreme case,
or some bug making it take much longer than needed.

Op do 8 apr. 2021 15:59 schreef Mario Bezzi <
subscriptions.mario.be...@gmail.com>:

> Thanks Jaap, I was not aware of this.
>
> To give you an idea, I just restarted Zim, and indexing kept a processor
> 100% busy for 13 minutes to come to an end.  It was nice if this could be
> avoided.
>
> Thank you,
> mario
>
> On 4/8/21 10:06 AM, Jaap Karssenberg wrote:
>
> The indexing is not used for searching alone, it is also needed to e.g.
> present the page tree in the side pane and to track links
>
> Op do 8 apr. 2021 09:34 schreef Mario Bezzi <
> subscriptions.mario.be...@gmail.com>:
>
>> Hello,
>>
>> I may be the only one, but with my quite large notebooks I do find the
>> search function impractical, and for this reason I never use it. Still,
>> when it starts, Zim goes crazy for a long time indexing, and I came to
>> the conclusion that this is normal.
>>
>> If this is the case, I would like to file a requirement to add the
>> ability to make indexing optional.
>>
>> Thank you,
>> mario
>>
>> ___
>> Mailing list: https://launchpad.net/~zim-wiki
>> Post to : zim-wiki@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~zim-wiki
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help   : https://help.launchpad.net/ListHelp

Re: [Zim-wiki] Making indexing optional ?

[Mario Bezzi]

Thanks Jaap, I was not aware of this.

To give you an idea, I just restarted Zim, and indexing kept a processor 
100% busy for 13 minutes to come to an end.  It was nice if this could 
be avoided.


Thank you,
mario

On 4/8/21 10:06 AM, Jaap Karssenberg wrote:
The indexing is not used for searching alone, it is also needed to 
e.g. present the page tree in the side pane and to track links


Op do 8 apr. 2021 09:34 schreef Mario Bezzi 
>:


Hello,

I may be the only one, but with my quite large notebooks I do find
the
search function impractical, and for this reason I never use it.
Still,
when it starts, Zim goes crazy for a long time indexing, and I
came to
the conclusion that this is normal.

If this is the case, I would like to file a requirement to add the
ability to make indexing optional.

Thank you,
mario

___
Mailing list: https://launchpad.net/~zim-wiki

Post to     : zim-wiki@lists.launchpad.net

Unsubscribe : https://launchpad.net/~zim-wiki

More help   : https://help.launchpad.net/ListHelp




___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help   : https://help.launchpad.net/ListHelp

Re: [Zim-wiki] Making indexing optional ?

[Jaap Karssenberg]

The indexing is not used for searching alone, it is also needed to e.g.
present the page tree in the side pane and to track links

Op do 8 apr. 2021 09:34 schreef Mario Bezzi <
subscriptions.mario.be...@gmail.com>:

> Hello,
>
> I may be the only one, but with my quite large notebooks I do find the
> search function impractical, and for this reason I never use it. Still,
> when it starts, Zim goes crazy for a long time indexing, and I came to
> the conclusion that this is normal.
>
> If this is the case, I would like to file a requirement to add the
> ability to make indexing optional.
>
> Thank you,
> mario
>
> ___
> Mailing list: https://launchpad.net/~zim-wiki
> Post to : zim-wiki@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~zim-wiki
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help   : https://help.launchpad.net/ListHelp

[Zim-wiki] Making indexing optional ?

[Mario Bezzi]

Hello,

I may be the only one, but with my quite large notebooks I do find the 
search function impractical, and for this reason I never use it. Still, 
when it starts, Zim goes crazy for a long time indexing, and I came to 
the conclusion that this is normal.


If this is the case, I would like to file a requirement to add the 
ability to make indexing optional.


Thank you,
mario

___
Mailing list: https://launchpad.net/~zim-wiki
Post to : zim-wiki@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zim-wiki
More help   : https://help.launchpad.net/ListHelp