Re: [tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-26 Thread PMario
On Thursday, September 26, 2019 at 10:21:14 AM UTC+2, TonyM wrote: ... > but just as tiddlywiki already has the type field for its own tiddler > types the best way for a designer to handle the example of classifying a > large number of tiddlers as words/Vorto is with a field be it containing a

Re: [tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-26 Thread TonyM
Folks, Great stuff here collaborating to resolve a performance issue. I hope also to emphasis something I stated early in this thread and in my own example of a high performing version. Tags have a particular ad hoc classification function. One which can be used so many ways, and is great

Re: [tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-25 Thread @TiddlyTweeter
Great to see something so clear! On Thursday, 26 September 2019 00:18:32 UTC+2, Arlen Beiler wrote: > > Ok, I found the offending line! And I double-checked that one line using > Date.now() and it literally does take six seconds. It varies by about 400 > ms from one time to the next, but

Re: [tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-25 Thread Arlen Beiler
When I step over the line it takes six seconds to show up for the next line. However, when I run a performance profile of the page and do exactly one action -- opening the control panel -- it still takes about 6 seconds but the performance profile shows thousands of refresh calls down the widget

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-25 Thread Jed Carty
It is used here: exports.getTiddlersWithTag = function(tag) { // Try to use the indexer var self = this, tagIndexer = this.getIndexer("TagIndexer"), results = tagIndexer && tagIndexer.subIndexers[3].lookup(tag); if(!results) { // If not available, perform a manual scan results =

Re: [tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-25 Thread Arlen Beiler
After stepping through conditional breakpoints and other things, the tag operator is indeed the result of the massive slowdown. [image: Screenshot 2019-09-25 13.46.38.png] The first line takes a long time to return, about 3 seconds per invocation. The rest of it is fairly quick. There is a

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-25 Thread 'Mark S.' via TiddlyWiki
Ah, but if your tiddler represented the *smallest semantic unit*, then it would be no trouble to add the Darcy tag at the top. And you would notice immediately that you already have #DarcyIsPrideful, #DarcyD'Man, and #DarcyHatethBottledWater. Later, when producing your final copy, you wouldn't

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-25 Thread @TiddlyTweeter
Its a very interesting issue, I think. I have NO idea if #hashtag in text field would be a PITA or not on performance in TW, not having tested it. I just think semantically it makes sense. I does well for the type of thing I write ... Mr #Darcy, cognisant of the plebesite, announces

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-25 Thread 'Mark S.' via TiddlyWiki
On Wednesday, September 25, 2019 at 3:25:05 AM UTC-7, @TiddlyTweeter wrote: > > > Personally I like the freeform taggery of tools like Twitter where > #hashtag is simply an in-text string to match. > > I was going to say that the hashtag would add a performance hit, but now I'm not so sure.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-25 Thread 'Mark S.' via TiddlyWiki
What is tidloid? It's not in the toolmap. I think autosave should already be off in EO-DICT. Thanks! On Tuesday, September 24, 2019 at 9:51:22 PM UTC-7, TonyM wrote: > > Mark, > > I used tidloid's fork a website and it opened but attempting to create a > new tiddler it froze. Presumably

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-25 Thread TonyM
TT Tiddlywiki allows you to do as you suggest out of the box. Just go ahead and use #tags in text. And filter on text contains or search. Perhaps a little experiment and publish this method is all that is needed. You could throw in some macros to make it easier or an editor toolbar button to

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-25 Thread @TiddlyTweeter
Mark S. wrote: > > ... I was thinking I would have to ditch tags, which is a shame since > Thomas' beautiful list-reveal makes > it so easy to add and remove tags. But contains:tags[Vorto] seems to work > without a noticeable performance hit. > TW has a kinda implicit philosophy of the

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-25 Thread @TiddlyTweeter
Mark S. I can use it on Android 8 successfully in Chrome (ONLINE version). Its slower than TonyM's previous but not unworkable. Loading would need, I think, a message so you know something is happening whilst is gears up. I'll comment later on opening a saved version on Android. TT Mark S.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-24 Thread TonyM
Mark, I used tidloid's fork a website and it opened but attempting to create a new tiddler it froze. Presumably saving. LG V20 Android 7.0 64GB internal memory turning of autosave made it work much better but touching the save changes icon took forever to demonstrate it was happening.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-24 Thread 'Mark S.' via TiddlyWiki
Now with 63,000 entrees. Eliminated dates and modifier/creator to save space. Comes in just under 10 megs. Crashed when attempting to save on tablet. Be interested to know if people are able to save on their small devices. https://marxsal.github.io/EODICT/EO-DICT.html On Thursday, September

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-24 Thread TonyM
Mat, > Mat > >> Tags should really be reserved for ad hoc unstructured labeling where it >> makes sense that one or more tags may be assigned. >> > > If this is so, then TW should feature more variants of "tags" (or whatever > the appropriate label would be then). For example, it is not

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-24 Thread Mat
TonyM wrote: > > Tags should really be reserved for ad hoc unstructured labeling where it > makes sense that one or more tags may be assigned. > If this is so, then TW should feature more variants of "tags" (or whatever the appropriate label would be then). For example, it is not trivial to

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-24 Thread TonyM
The truth is if a large number of items have one tag it makes sense to me not to use a tag. I would instead use a field. Such as vorto and use has:field[vorto] where only the existence of the field is required. Then if I had a set of different types I may use a field such as objectytpe with the

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-23 Thread Mat
Mark S. wrote: > > [...] contains:tags[Vorto] seems to work without a noticeable performance > hit. > Ah, great! I must assume this raises the interesting question whether the tags operator can/should be modified to behave like contains:tags instead, perhaps even replace the tags op all

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-23 Thread 'Mark S.' via TiddlyWiki
If anyone is curious, here is the work in progress (only 36k -- half the target lexicon) https://marxsal.github.io/EODICT/EO-DICT.html -- You received this message because you are subscribed to the Google Groups "TiddlyWiki" group. To unsubscribe from this group and stop receiving emails from

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-23 Thread 'Mark S.' via TiddlyWiki
Hi Mat, That was a very good thought. Previously I was thinking I would have to ditch tags, which is a shame since Thomas' beautiful list-reveal makes it so easy to add and remove tags. But contains:tags[Vorto] seems to work without a noticeable performance hit. Thanks! On Sunday, September

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-23 Thread A Gloom
EDIT: Only when that tiddler is open in the sidebar, I get what Mark gets-- > everything-- typing, clearing the search box, hiding/reveraling the > sidebar, switching sidebar tabs has about a 5 second lag. > Only when the tiddler is open in the *(sidebar)* story river *(not sidebar) -- You

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-23 Thread A Gloom
I have created a similiar effect in a much smaller but more complex wiki single file TW, vers 5.1.20, 4.3 Mb file size, ~1700 tiddlers, latest FF, 16 Gb RAM I already get about a one sec delay when typing in the tag selection field once I went over 500 tags but otherwise the wiki doesn't show

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-22 Thread TonyM
Possibly related To tag search performance. Unless you use sort[] list of tiddlers with a given tag will be sorted in the order they are listed in a list field. I would assume thus tiddlywiki needs to look for list field existence on any tag. Also any tag pill for a tag, one exists for every

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-22 Thread Mat
Did you try [contains:tags[Vorto]] ? <:-) -- You received this message because you are subscribed to the Google Groups "TiddlyWiki" group. To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+unsubscr...@googlegroups.com. To view this discussion on the

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-22 Thread 'Mark S.' via TiddlyWiki
Seems like the answer might be a separate tag filter operator for use on data with high use counts. Then the developer could decide when it was most appropriate to use whichever tag filter operator. On Sunday, September 22, 2019 at 4:13:40 AM UTC-7, Jed Carty wrote: > > Each filter operator

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-22 Thread Jed Carty
Each filter operator only works on the output of the previous operator, so the order matters with later operators in a run generally working with fewer inputs than earlier operators. To my knowledge none of the operators do anything in parallel, that may be difficult to set up because

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-22 Thread Mat
Mark, that might be a huge step forward. Here is a brain dump of hypotheses and thoughts from someone who doesn't really know what he is talking about. You will, for sure, think "what is this fool twaddling about?" but while my ignorance is typically a weakness it occasionally brings an

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-22 Thread TonyM
Mark, Perhaps try these test in a 5.1.19 Wiki given the additional indexing to improve performance was introduced in 5.1.20 and if there was any change that could have had an unexpected impact in this area I think it could be those changes. Keeping in mind the tags field potentially contains

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-21 Thread 'Mark S.' via TiddlyWiki
It appears that having a search, list, or list-links that uses a *tag operator* running in any currently open tiddler will create a performance problem EVEN if the tag operator is applied *AFTER *a previous search operator. So this

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-21 Thread 'Mark S.' via TiddlyWiki
Using "[all[current]has[en]]" did not show any significant improvement. On Saturday, September 21, 2019 at 12:59:28 PM UTC-7, Mark S. wrote: > > Hi Mat, > > I've already tried [tag[Vorto]], if that's what you meant > by the first part. Alas, it didn't help. > > Testing for foofield[Vorto] is a

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-21 Thread 'Mark S.' via TiddlyWiki
Hi Mat, I've already tried [tag[Vorto]], if that's what you meant by the first part. Alas, it didn't help. Testing for foofield[Vorto] is a good idea. But I'd have to first modify 36,000 tiddlers. So I'm kind of hoping someone will pop up with "Oh, that's a known problem with template tiddlers

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-21 Thread Mat
Mark S. wrote: > > > <$list filter="[all[current]tag[Vorto]]"> > Eng: <$edit-text field="en" tag="textarea" rows=2 class="vorto-edit" /> > > What happens if you exchange the operators for "synonyms" like instead of "all[current]" do "[[tiddlertitle]tag[Vorto]]" or

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-21 Thread 'Mark S.' via TiddlyWiki
Hi Birthe, Tony -- everyone! You've all been very helpful! I think I know *what *is causing the problem -- but I don't know *why*. There's a view template (WordTemplate) used to expose the en field. The contents are typical for templates: <$list filter="[all[current]tag[Vorto]]"> Eng:

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-20 Thread 'Mark S.' via TiddlyWiki
It's slow even if non-searching tiddlers are all that is rendered. It's slow with only 6 other tabs open. It's slow even if FF has just been opened. It's slow even though instrumentation is off. It's slow even if doing a single-field match. It's not just searching that's slow. EVERYTHING you type

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-20 Thread Birthe C
Hi Mark S, You are around the size 10 megs where I usually get into trouble. Recent i got a "new" computer with more ram, it did help a little but really not enough for it to be fun to work with these wikies.. Former I sometimes used Firefox, Windows7 and found it slower than Firefox, Linux

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-20 Thread @TiddlyTweeter
Mark S. wrote: > > At the moment it seems like I need to start over, using a data > dictionary instead of > tiddlers for storage. > I'd be very interested to know, if you go ahead with a data dictionary version, what the performance is like! Best wishes TT -- You received this message

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-19 Thread Mohammad
TW-Scripts has 4000+ tiddlers including all shadow tiddlers. I am on 5.1.21, Firefox 69, Win 10 I see noticeable response time comparing to empty.html --Mohammad On Friday, September 20, 2019 at 8:49:37 AM UTC+4:30, Mark S. wrote: > > My project may have been over ambitious. But I thought

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-19 Thread TonyM
Post Script If you a doing word searches, ie words as tiddler titles, not words in phrases or in the test fields changing the search not to look only in titles (not in text) or even search first by prefix could improve search outcomes a lot. Regards Tony On Friday, September 20, 2019 at

[tw5] Re: Hitting a barrier at 36,000 tiddlers

2019-09-19 Thread TonyM
Mark, To fully address your experience consider if you can show or share the wiki. Alternativy share some test data of an equivalent size. Also consider it from a "what must it render perspective". If for example the tiddlers were all listed in the table of contents and this is displayed in