Re: [tw5] Re: Hitting a barrier at 36,000 tiddlers

[PMario]

On Thursday, September 26, 2019 at 10:21:14 AM UTC+2, TonyM wrote:
 ...

> but just as tiddlywiki already has the type field for its own tiddler 
> types the best way for a designer to handle the example of classifying a 
> large number of tiddlers as words/Vorto is with a field be it containing a 
> tiddler-type value or just an if exist "word" field. Why because it does 
> not use a heavily flexible mechanism for tags to do a simpler job, and in 
> large quantities this makes a difference.
>

I think using the tiddler-type field would be a solution here, since it 
wouldn't need to use the 

<$list filter="[all[current]tag[xxx]]">    *hack*, to create a custom 
ViewTemplate or / and EditTemplate. So the "WordTemplate" could be much 
simpler.

The TW tiddler-type field is used to specify a MIME-type 
. It is possible to create your 
own "vendor specific" (like text/vnd.tiddlywiki) or "personal" 
(text/prs.dictionary) types, but you need to follow some rules. 

Personal or vanity tree
>
> The personal or vanity tree includes media types associated with non 
> publicly available products or experimental media types. It uses the prs. 
> tree prefix: 
> type "/" "prs." subtype ["+" suffix] *[";" parameter] 
>
> Examples: audio/prs.sid, image/prs.btif. 
>
> from Wikipedia
>

So a text/prs.dict.word;language=en-EN or something similar, would be a 
possibility to work with a specification as described in the OP. 

The problem is, that we don't have good end-user support for content like 
this in TW at the moment. 

just some thoughts. 
-mario





-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/18bfb65f-8267-4ba3-acfc-a2aa4e86251d%40googlegroups.com.

Re: [tw5] Re: Hitting a barrier at 36,000 tiddlers

[TonyM]

Folks,

Great stuff here collaborating to resolve a performance issue.

I hope also to emphasis something I stated early in this thread and in my 
own example of a high performing version. Tags have a particular ad hoc 
classification function. One which can be used so many ways, and is great 
for small wikis, because you can use the same mechanism for many purposes. 
Of course with lots of different tags the "tag space becomes polluted or 
messy" It is possible to have multiple tag like fields, see  Generic Tags 
Plugin  but just as 
tiddlywiki already has the type field for its own tiddler types the best 
way for a designer to handle the example of classifying a large number of 
tiddlers as words/Vorto is with a field be it containing a tiddler-type 
value or just an if exist "word" field. Why because it does not use a 
heavily flexible mechanism for tags to do a simpler job, and in large 
quantities this makes a difference.

Not with standing I have what I consider a good technical and design reason 
for avoiding tags in this case, it is true that this would not occur to 
most users/designer. As a result perhaps we can encourage the use of a 
field such as user-type or object-type which contains a single value from a 
number of possible values defined by a user. We could even support this by 
providing a drop down that lets you select from *each value* found in this 
field anywhere in the wiki (along with edit text to enter new ones). Its 
about providing a gentle push towards a design pattern that will serve them 
well.

Regards
Tony



On Thursday, September 26, 2019 at 8:56:16 AM UTC+10, Arlen Beiler wrote:
>
> I opened an issue on GitHub: 
> https://github.com/Jermolene/TiddlyWiki5/issues/4275
>
> On Wed, Sep 25, 2019 at 6:47 PM Arlen Beiler  > wrote:
>
>> I've been studying this function for a half hour (yes, sometimes that's 
>> how I work -- inefficient, I know). Anyway, it suddenly occurred to me that 
>>
>> *if* new position is -1 *then* current position is useless, 
>>
>>- *because* right now *if* new position is -1, *then* it is set to 
>>current position
>>- *immediately after that* it checks *if they are equal* and only 
>>replaces them *if they are not*. 
>>
>> So if new position was set to -1, then it will be set to current 
>> position, and therefore is equal to it, and therefore prevents a splice 
>> from happening. So we could totally optimize this if new position is still 
>> -1 after the first group of if statements and just exit the function if 
>> there is nothing to do without looking up the current position. 
>>
>> if(newPos !== -1) {
>>   var currPos = titles.indexOf(title);
>>   if(currPos !== -1) {
>> titles.splice(currPos, 1);
>> if(newPos >= currPos) newPos--;
>> titles.splice(newPos, 0, title);
>>   }
>> }
>>
>> Of course, you still can't use list-before and list-after on 36000 
>> tiddlers, but who has ever done that?
>>
>> On Wed, Sep 25, 2019 at 6:18 PM Arlen Beiler > > wrote:
>>
>>> Ok, I found the offending line! And I double-checked that one line using 
>>> Date.now() and it literally does take six seconds. It varies by about 400 
>>> ms from one time to the next, but that's completely accurate! Note that the 
>>> line is executed many times during the sort and that is the total execution 
>>> time for that line for one sort operation. 
>>>
>>> [image: Screenshot 2019-09-25 18.02.35.png]
>>>
>>> On Wed, Sep 25, 2019 at 5:58 PM Arlen Beiler >> > wrote:
>>>
 When I step over the line it takes six seconds to show up for the next 
 line. However, when I run a performance profile of the page and do exactly 
 one action -- opening the control panel -- it still takes about 6 seconds 
 but the performance profile shows thousands of refresh calls down the 
 widget tree one after the next for six seconds. It blames the whole thing 
 on sortByList, and specifically the replaceItem function inside it, but 
 I'm 
 still investigating it.

 On Wed, Sep 25, 2019 at 4:12 PM Jed Carty >>> > wrote:

> It is used here:
>
> exports.getTiddlersWithTag = function(tag) {
>  // Try to use the indexer
>  var self = this,
>  tagIndexer = this.getIndexer("TagIndexer"),
>  results = tagIndexer && tagIndexer.subIndexers[3].lookup(tag);
>  if(!results) {
>  // If not available, perform a manual scan
>  results = this.getGlobalCache("taglist-" + tag,function() {
>  var tagmap = self.getTagMap();
>  return self.sortByList(tagmap[tag],tag);
>  });
>  }
>  return results;
> };
>
> which caches the results to make it faster afterward, this works well 
> for when a tag isn't on too many tiddlers, but I have no idea what 'too 
> many' is here.
>
> I haven't dug too deeply into the cache mechanism and there may be 
> some maximum size after which it won't cache results, if that is the 

Re: [tw5] Re: Hitting a barrier at 36,000 tiddlers

[@TiddlyTweeter]

Great to see something so clear! 

On Thursday, 26 September 2019 00:18:32 UTC+2, Arlen Beiler wrote:
>
> Ok, I found the offending line! And I double-checked that one line using 
> Date.now() and it literally does take six seconds. It varies by about 400 
> ms from one time to the next, but that's completely accurate! Note that the 
> line is executed many times during the sort and that is the total execution 
> time for that line for one sort operation. 
>
> [image: Screenshot 2019-09-25 18.02.35.png]
>
> On Wed, Sep 25, 2019 at 5:58 PM Arlen Beiler  > wrote:
>
>> When I step over the line it takes six seconds to show up for the next 
>> line. However, when I run a performance profile of the page and do exactly 
>> one action -- opening the control panel -- it still takes about 6 seconds 
>> but the performance profile shows thousands of refresh calls down the 
>> widget tree one after the next for six seconds. It blames the whole thing 
>> on sortByList, and specifically the replaceItem function inside it, but I'm 
>> still investigating it.
>>
>> On Wed, Sep 25, 2019 at 4:12 PM Jed Carty > > wrote:
>>
>>> It is used here:
>>>
>>> exports.getTiddlersWithTag = function(tag) {
>>>  // Try to use the indexer
>>>  var self = this,
>>>  tagIndexer = this.getIndexer("TagIndexer"),
>>>  results = tagIndexer && tagIndexer.subIndexers[3].lookup(tag);
>>>  if(!results) {
>>>  // If not available, perform a manual scan
>>>  results = this.getGlobalCache("taglist-" + tag,function() {
>>>  var tagmap = self.getTagMap();
>>>  return self.sortByList(tagmap[tag],tag);
>>>  });
>>>  }
>>>  return results;
>>> };
>>>
>>> which caches the results to make it faster afterward, this works well 
>>> for when a tag isn't on too many tiddlers, but I have no idea what 'too 
>>> many' is here.
>>>
>>> I haven't dug too deeply into the cache mechanism and there may be some 
>>> maximum size after which it won't cache results, if that is the case than 
>>> it may have to build the results every time the tag is looked up instead of 
>>> being able to use the cached result.
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "TiddlyWiki" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to tiddl...@googlegroups.com .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/tiddlywiki/160dda32-67cc-4078-b76a-3e2408239d99%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/c68e891f-f2d3-4c68-881e-8c4438e68f87%40googlegroups.com.

Re: [tw5] Re: Hitting a barrier at 36,000 tiddlers

[Arlen Beiler]

When I step over the line it takes six seconds to show up for the next
line. However, when I run a performance profile of the page and do exactly
one action -- opening the control panel -- it still takes about 6 seconds
but the performance profile shows thousands of refresh calls down the
widget tree one after the next for six seconds. It blames the whole thing
on sortByList, and specifically the replaceItem function inside it, but I'm
still investigating it.

On Wed, Sep 25, 2019 at 4:12 PM Jed Carty  wrote:

> It is used here:
>
> exports.getTiddlersWithTag = function(tag) {
>  // Try to use the indexer
>  var self = this,
>  tagIndexer = this.getIndexer("TagIndexer"),
>  results = tagIndexer && tagIndexer.subIndexers[3].lookup(tag);
>  if(!results) {
>  // If not available, perform a manual scan
>  results = this.getGlobalCache("taglist-" + tag,function() {
>  var tagmap = self.getTagMap();
>  return self.sortByList(tagmap[tag],tag);
>  });
>  }
>  return results;
> };
>
> which caches the results to make it faster afterward, this works well for
> when a tag isn't on too many tiddlers, but I have no idea what 'too many'
> is here.
>
> I haven't dug too deeply into the cache mechanism and there may be some
> maximum size after which it won't cache results, if that is the case than
> it may have to build the results every time the tag is looked up instead of
> being able to use the cached result.
>
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tiddlywiki+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/tiddlywiki/160dda32-67cc-4078-b76a-3e2408239d99%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/CAJ1vdSSz%3DsWcnyK_htJMHyPguTOPEPneS8%2B%3DA77o25qb1q7GEg%40mail.gmail.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[Jed Carty]

It is used here:

exports.getTiddlersWithTag = function(tag) {
 // Try to use the indexer
 var self = this,
 tagIndexer = this.getIndexer("TagIndexer"),
 results = tagIndexer && tagIndexer.subIndexers[3].lookup(tag);
 if(!results) {
 // If not available, perform a manual scan
 results = this.getGlobalCache("taglist-" + tag,function() {
 var tagmap = self.getTagMap();
 return self.sortByList(tagmap[tag],tag);
 });
 }
 return results;
};

which caches the results to make it faster afterward, this works well for 
when a tag isn't on too many tiddlers, but I have no idea what 'too many' 
is here.

I haven't dug too deeply into the cache mechanism and there may be some 
maximum size after which it won't cache results, if that is the case than 
it may have to build the results every time the tag is looked up instead of 
being able to use the cached result.

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/160dda32-67cc-4078-b76a-3e2408239d99%40googlegroups.com.

Re: [tw5] Re: Hitting a barrier at 36,000 tiddlers

[Arlen Beiler]

After stepping through conditional breakpoints and other things, the tag
operator is indeed the result of the massive slowdown.

[image: Screenshot 2019-09-25 13.46.38.png]

The first line takes a long time to return, about 3 seconds per invocation.
The rest of it is fairly quick. There is a source.byTag operation right
before it that would be much quicker but I don't know where it comes from
or what uses it. I guess that would take some investigating. But this
basically means that using the tag operator for any tags which have many
tiddlers is basically out of the question. So everyone's basically on the
right track it seems.

On Wed, Sep 25, 2019 at 12:44 PM 'Mark S.' via TiddlyWiki <
tiddlywiki@googlegroups.com> wrote:

> Ah, but if your tiddler represented the *smallest semantic unit*, then it
> would be no trouble to add the Darcy tag at the top.
> And you would notice immediately that you already have #DarcyIsPrideful,
> #DarcyD'Man, and #DarcyHatethBottledWater.
>
> Later, when producing your final copy, you wouldn't have to scrape out all
> the little # symbols, like pulling burrs off
> a dog that's been wandering too long in the 100 acre forest.
>
> “If I ever go looking for my heart's desire again, I won't look any
>> further than my own back yard.
>> Because if it isn't there, I never really lost it to begin with… "
>>
>
> Ok, a quote from P would be better, but I only read it once and never
> understood what the zombies were doing ...
>
>
> On Wednesday, September 25, 2019 at 7:37:12 AM UTC-7, @TiddlyTweeter wrote:
>>
>> Its a very interesting issue, I think.
>>
>> I have NO idea if #hashtag in text field would be a PITA or not on
>> performance in TW, not having tested it.
>>
>> I just think semantically it makes sense. I does well for the type of
>> thing I write ...
>>
>> Mr #Darcy, cognisant of the plebesite,  announces #BottledWater should be
>>> banned.
>>
>>
>> You get the idea.
>>
>> I SIMPLY like to write and sort out an order later.
>>
>> TT
>>
>>
>>
>> On Wednesday, 25 September 2019 16:24:01 UTC+2, Mark S. wrote:
>>>
>>>
>>>
>>> On Wednesday, September 25, 2019 at 3:25:05 AM UTC-7, @TiddlyTweeter
>>> wrote:


 Personally I like the freeform taggery of tools like Twitter where
 #hashtag is simply an in-text string to match.


>>> I was going to say that the hashtag would add a performance hit, but now
>>> I'm not so sure. Using regular tags
>>> actually created a performance hit. But one problem is that people, left
>>> to themselves, will make multiple
>>> variations of the same tag.
>>>
>>> #myhashtag #MY-hash #TAG-Hash #hash-tag-de-mio
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tiddlywiki+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/tiddlywiki/c1878946-113b-49ac-b66c-6a38c6f1f29e%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/CAJ1vdSScH6QUzdaV_eEHkF6yEYwPVofBndFwMXjPLt%3DRYOhrkw%40mail.gmail.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

['Mark S.' via TiddlyWiki]

Ah, but if your tiddler represented the *smallest semantic unit*, then it 
would be no trouble to add the Darcy tag at the top.
And you would notice immediately that you already have #DarcyIsPrideful, 
#DarcyD'Man, and #DarcyHatethBottledWater.

Later, when producing your final copy, you wouldn't have to scrape out all 
the little # symbols, like pulling burrs off
a dog that's been wandering too long in the 100 acre forest.

“If I ever go looking for my heart's desire again, I won't look any further 
> than my own back yard. 
> Because if it isn't there, I never really lost it to begin with… "
>

Ok, a quote from P would be better, but I only read it once and never 
understood what the zombies were doing ...


On Wednesday, September 25, 2019 at 7:37:12 AM UTC-7, @TiddlyTweeter wrote:
>
> Its a very interesting issue, I think.
>
> I have NO idea if #hashtag in text field would be a PITA or not on 
> performance in TW, not having tested it.
>
> I just think semantically it makes sense. I does well for the type of 
> thing I write ...
>
> Mr #Darcy, cognisant of the plebesite,  announces #BottledWater should be 
>> banned.
>
>
> You get the idea.
>
> I SIMPLY like to write and sort out an order later.
>
> TT 
>
>
>
> On Wednesday, 25 September 2019 16:24:01 UTC+2, Mark S. wrote:
>>
>>
>>
>> On Wednesday, September 25, 2019 at 3:25:05 AM UTC-7, @TiddlyTweeter 
>> wrote:
>>>
>>>
>>> Personally I like the freeform taggery of tools like Twitter where 
>>> #hashtag is simply an in-text string to match.
>>>
>>>
>> I was going to say that the hashtag would add a performance hit, but now 
>> I'm not so sure. Using regular tags
>> actually created a performance hit. But one problem is that people, left 
>> to themselves, will make multiple
>> variations of the same tag. 
>>
>> #myhashtag #MY-hash #TAG-Hash #hash-tag-de-mio 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/c1878946-113b-49ac-b66c-6a38c6f1f29e%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[@TiddlyTweeter]

Its a very interesting issue, I think.

I have NO idea if #hashtag in text field would be a PITA or not on 
performance in TW, not having tested it.

I just think semantically it makes sense. I does well for the type of thing 
I write ...

Mr #Darcy, cognisant of the plebesite,  announces #BottledWater should be 
> banned.


You get the idea.

I SIMPLY like to write and sort out an order later.

TT 



On Wednesday, 25 September 2019 16:24:01 UTC+2, Mark S. wrote:
>
>
>
> On Wednesday, September 25, 2019 at 3:25:05 AM UTC-7, @TiddlyTweeter wrote:
>>
>>
>> Personally I like the freeform taggery of tools like Twitter where 
>> #hashtag is simply an in-text string to match.
>>
>>
> I was going to say that the hashtag would add a performance hit, but now 
> I'm not so sure. Using regular tags
> actually created a performance hit. But one problem is that people, left 
> to themselves, will make multiple
> variations of the same tag. 
>
> #myhashtag #MY-hash #TAG-Hash #hash-tag-de-mio 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/6dc1047f-4d49-4b35-97d3-e4cf71cf1a57%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

['Mark S.' via TiddlyWiki]

On Wednesday, September 25, 2019 at 3:25:05 AM UTC-7, @TiddlyTweeter wrote:
>
>
> Personally I like the freeform taggery of tools like Twitter where 
> #hashtag is simply an in-text string to match.
>
>
I was going to say that the hashtag would add a performance hit, but now 
I'm not so sure. Using regular tags
actually created a performance hit. But one problem is that people, left to 
themselves, will make multiple
variations of the same tag. 

#myhashtag #MY-hash #TAG-Hash #hash-tag-de-mio 

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/3225b535-f77b-4dcc-9650-8a2715b64c83%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

['Mark S.' via TiddlyWiki]

What is tidloid? It's not in the toolmap. I think autosave should already 
be off in EO-DICT.

Thanks!

On Tuesday, September 24, 2019 at 9:51:22 PM UTC-7, TonyM wrote:
>
> Mark,
>
> I used tidloid's fork a website and it opened but attempting to create a 
> new tiddler it froze. Presumably saving.
>
> LG V20 Android 7.0 64GB internal memory
>
> turning of autosave made it work much better but touching the save changes 
> icon took forever to demonstrate it was happening.
> Reloading from tiddloid menu takes quite some time but it seems to work
>
> Regards
> Tony
>
>
>
>
> On Wednesday, September 25, 2019 at 2:24:02 PM UTC+10, Mark S. wrote:
>>
>>
>> Now with 63,000 entrees. Eliminated dates and modifier/creator to save 
>> space. Comes in just under 10 megs. 
>> Crashed when attempting to save on tablet. Be interested to know if 
>> people are able to save on their small
>> devices.
>>
>> https://marxsal.github.io/EODICT/EO-DICT.html
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/89d7def9-920e-473d-84d5-d3f47671e70d%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[TonyM]

TT

Tiddlywiki allows you to do as you suggest out of the box. Just go ahead and 
use #tags in text. And filter on text contains or search. Perhaps a little 
experiment and publish this method is all that is needed. You could throw in 
some macros to make it easier or an editor toolbar button to insert existing 
hash tags. Ideally you could register new hashtags in a data tiddler so you 
know what you can search for or log their use.

Of course if the wiki has a lot of text or tiddler it may be wise to move such 
hash tags into a tag like field. 

Would it be ok to use $:/#tagname because then you could have the option of 
creating a tiddler and using it as a tag. You could then have a view template 
that with [all[current]prefix[$:/#]] displaying hash tag tools when you open a 
$:/#tagname tiddler. References etc will also work with this.

Imagin if you could make # or /# a new namespace like $:/ is

Regards
Tony

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/c136c451-e8c7-445b-845d-dfcf2de93d4b%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[@TiddlyTweeter]

Mark S. wrote:
>
> ... I was thinking I would have to ditch tags, which is a shame since 
> Thomas' beautiful list-reveal makes
> it so easy to add and remove tags. But contains:tags[Vorto] seems to work 
> without a noticeable performance hit.
>

TW has a kinda implicit philosophy of the *Uber-Role of Tags*. Tags are 
doing about 85 jobs.

They are for structure. They are for content signalling. They have 
sophisticated parsing.

Personally I like the freeform taggery of tools like Twitter where #hashtag 
is simply an in-text string to match.

I think my underlying question is "are we neglecting the text field?"

Hope this is not too off-topic.

Best wishes
TT

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/d1a46567-b7ce-4f9c-b29e-ef46e782ae33%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[@TiddlyTweeter]

Mark S.

I can use it on Android 8 successfully in Chrome (ONLINE version). Its 
slower than TonyM's previous but not unworkable. Loading would need, I 
think, a message so you know something is happening whilst is gears up.

I'll comment later on opening a saved version on Android.

TT



Mark S. wrote:
>
> Be interested to know if people are able to save on their small
> devices.
>
> https://marxsal.github.io/EODICT/EO-DICT.html
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/5a136df6-d038-466a-9f25-3e190715cb65%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[TonyM]

Mark,

I used tidloid's fork a website and it opened but attempting to create a 
new tiddler it froze. Presumably saving.

LG V20 Android 7.0 64GB internal memory

turning of autosave made it work much better but touching the save changes 
icon took forever to demonstrate it was happening.
Reloading from tiddloid menu takes quite some time but it seems to work

Regards
Tony




On Wednesday, September 25, 2019 at 2:24:02 PM UTC+10, Mark S. wrote:
>
>
> Now with 63,000 entrees. Eliminated dates and modifier/creator to save 
> space. Comes in just under 10 megs. 
> Crashed when attempting to save on tablet. Be interested to know if people 
> are able to save on their small
> devices.
>
> https://marxsal.github.io/EODICT/EO-DICT.html
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/6b483679-a6ed-4985-aa76-de92baa82527%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

['Mark S.' via TiddlyWiki]

Now with 63,000 entrees. Eliminated dates and modifier/creator to save 
space. Comes in just under 10 megs. 
Crashed when attempting to save on tablet. Be interested to know if people 
are able to save on their small
devices.

https://marxsal.github.io/EODICT/EO-DICT.html


On Thursday, September 19, 2019 at 9:19:37 PM UTC-7, Mark S. wrote:
>
> My project may have been over ambitious. But I thought there were going to 
> be some 
> improvements in 5.1.20+ that would allow for larger TW files. I currently 
> have a TW file
> with about 36,000 short dictionary definitions. Each tiddler consists only 
> of a title
> and a field, EN, with usually only 3 or four words in it. The overall size 
> is only 10 megs.
>
> Overall, the performance is impracticably slow. Ten seconds to open a 
> tiddler. 20 seconds or more
> to do a simple search box search. As much as 25 seconds or more to close 
> out tiddlers that
> need to render a custom search. In most cases, you have to type blind 
> because the
> keystroke refresh is also incredibly slow.
>
> I'm wondering if the performance improvements actually made it into the 
> core, or if
> my expectations were too high. Actually, the performance noticeably 
> declined
> after the first 12,000 entries. 
>
> At the moment it seems like I need to start over, using a data dictionary 
> instead of
> tiddlers for storage. 
>
> Firefox, Windows 7, TW 5.1.21
>
> Thanks!
> Mark
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/85b5fc42-57b7-46a3-a6ab-924fe2a47a0a%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[TonyM]

Mat, 
 

> Mat
>
>> Tags should really be reserved for ad hoc unstructured labeling where it 
>> makes sense that one or more tags may be assigned. 
>>
>  
> If this is so, then TW should feature more variants of "tags" (or whatever 
> the appropriate label would be then). For example, it is not trivial to get 
> "myfield:Vorto" to display as a tag pill, which you might want it to.
>

This is a useful possible improvement, but of course they will not be tag 
pills but field pills. Of course it depends on what you want from the pill. 
I like the idea that the field pill drop down could present possible values 
for the field and a click will populate that field on the current tiddler. 
Ideally one could set different filter on each field pill from canned 
values to the each of values found in the field through out the wiki. I am 
not so good at writing popups yet.
 

>
> Because tags are so versatile I am not surprised they are used less 
>> appropriately.
>
>
> Well, obviously Mark is an absolute TW expert so I'd say your 
> interpretation of how tags are to be used is merely an opinion. The 
> question is, again, if the system meets our needs. 
>

This is not a criticism of marks design but a suggestion already made with 
others for performance, it also highlights this will be common for all 
tiddlywiki users because the tag system is so versatile. 
 

>
> Fortunately it is easy to transfer values between tags and fields
>
>
> There's certainly no UI for it so it is not trivial. How do you go about 
> it?
>
> <:-)
>

Again a good suggestion. A UI for this. Some tools provide this (Mohammads 
commander?) but this is what one does

   - Create a List widget that creates the list of tiddlers you want to 
   change
   - Place the actions (Widgets) you want performed inside the list
   - Wrap all of it in a button.

Here is one I prepared
;Filter: <$edit-text tiddler="$:/temp/bulk/filter" tag=input/>
;Actions: 
<$tiddler tiddler="$:/temp/bulk/actions">
<$transclude tiddler="$:/core/ui/EditTemplate/body/editor"/>

:`Warning - most actions may be irreversible` make backups
://consider a two step process for intermediate results//

Tiddlers to be actioned - click to open/close<
/summary>

<$list filter={{$:/temp/bulk/filter}}>







<$button>
<$list filter={{$:/temp/bulk/filter}}>
{{||$:/temp/bulk/actions}}

Apply actions to selected tiddlers


 Perhaps a version of this with the ability to inserts filters (From 
advanced search) and actions from a drop down should be provisioned?

Regards
Tony

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/d81996ae-5886-4767-b8a7-70bd570b4e8b%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[Mat]

TonyM wrote:
>
> Tags should really be reserved for ad hoc unstructured labeling where it 
> makes sense that one or more tags may be assigned. 
>
 
If this is so, then TW should feature more variants of "tags" (or whatever 
the appropriate label would be then). For example, it is not trivial to get 
"myfield:Vorto" to display as a tag pill, which you might want it to.

Because tags are so versatile I am not surprised they are used less 
> appropriately.


Well, obviously Mark is an absolute TW expert so I'd say your 
interpretation of how tags are to be used is merely an opinion. The 
question is, again, if the system meets our needs. 

Fortunately it is easy to transfer values between tags and fields


There's certainly no UI for it so it is not trivial. How do you go about it?

<:-)

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/aed2c13f-66ee-4ce6-affc-f7e8b71fc06f%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[TonyM]

The truth is if a large number of items have one tag it makes sense to me not 
to use a tag. I would instead use a field. Such as vorto and use 
has:field[vorto] where only the existence of the field is required. Then if I 
had a set of different types I may use a field such as objectytpe with the 
value vorto thus select with objecttype[vorto] where object type only ever has 
one value of a limited set.

Tags should really be reserved for ad hoc unstructured labeling where it makes 
sense that one or more tags may be assigned. Fortunately it is easy to transfer 
values between tags and fields


Because tags are so versatile I am not surprised they are used less 
appropriately.

Before I use a tag a field etc I always ask my self about the nature of the 
value I am setting. I Actualy do this intuitively so it may be a job to list 
the conditions I use to choose.
Regards
Tony 



-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/3d233be5-d9e5-4514-ba01-bfb8ac974a97%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[Mat]

Mark S. wrote:
>
> [...] contains:tags[Vorto] seems to work without a noticeable performance 
> hit.
>

Ah, great! I must assume this raises the interesting question whether the 
tags operator can/should be modified to behave like contains:tags instead, 
perhaps even replace the tags op all together? And what causes tags to 
misbehave in the first place? (BTW, is @Jermolene following this 
discussion?) 

<:-)

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/28c97d2f-5e33-46ae-bf74-8e6ff762a406%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

['Mark S.' via TiddlyWiki]

If anyone is curious, here is the work in progress (only 36k -- half the 
target lexicon)

https://marxsal.github.io/EODICT/EO-DICT.html

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/de11acaa-a4af-4088-892e-177cf73a6667%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

['Mark S.' via TiddlyWiki]

Hi Mat,

That was a very good thought. Previously I was thinking I would have to 
ditch tags, which is a shame since Thomas' beautiful list-reveal makes
it so easy to add and remove tags. But contains:tags[Vorto] seems to work 
without a noticeable performance hit.

Thanks!

On Sunday, September 22, 2019 at 9:43:10 PM UTC-7, Mat wrote:
>
> Did you try [contains:tags[Vorto]] ?
>
> <:-)
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/3064fa61-51dc-4f85-be8c-a4e8cf591438%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[A Gloom]

EDIT:

Only when that tiddler is open in the sidebar, I get what Mark gets-- 
> everything-- typing, clearing the search box, hiding/reveraling the 
> sidebar, switching sidebar tabs has about a 5 second lag.
>

Only when the tiddler is open in the *(sidebar)* story river *(not sidebar)

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/2e680ca5-3bb8-4baf-a7d5-3f7ceef3abd2%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[A Gloom]

I have created a similiar effect in a much smaller but more complex wiki

single file TW, vers 5.1.20, 4.3 Mb file size, ~1700 tiddlers, latest FF, 
16 Gb RAM

I already get about a one sec delay when typing in the tag selection field 
once I went over 500 tags but otherwise the wiki doesn't show much 
lagging-- except one tiddler full of filters, lists and transclusions.  
Only when that tiddler is open in the sidebar, I get what Mark gets-- 
everything-- typing, clearing the search box, hiding/reveraling the 
sidebar, switching sidebar tabs has about a 5 second lag.

It uses a bunch of view widfets but then it uses a lot of many other things 
(including mixed wikitext/html and in tiddler styling) so narrowing down 
the culprit has been very time consuming.  Need to use devtools but dev 
tools unusable for me to do serious troubleshooting with.

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/fba0950b-d0f4-4967-9158-a06bf5919695%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[TonyM]

Possibly related To tag search performance.

Unless you use sort[] list of tiddlers with a given tag will be sorted in 
the order they are listed in a list field. I would assume thus tiddlywiki 
needs to look for list field existence on any tag.

Also any tag pill for a tag, one exists for every Word in the dictionary 
tagged with Vorto if you click on the tag pill it will list all 36,000 
words so tagged in the tag pill drop down. Are there dropdown populated 
only once opened or before hand. If before hand every tag pill with contain 
a 36,000 item drop down.

I note that in my high performing wiki of ESPDIC 
https://tiddlywiki.psat.com.au/ESPDIC/ that the More Tags field does not 
include Vorto, and there is no vorto tiddler. 

Regards
Tony


On Monday, September 23, 2019 at 2:43:10 PM UTC+10, Mat wrote:
>
> Did you try [contains:tags[Vorto]] ?
>
> <:-)
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/75c4e96c-5393-43f9-a2cf-e3d18c37ee48%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[Mat]

Did you try [contains:tags[Vorto]] ?

<:-)

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/65ad745e-51a0-4f20-bac8-d9fce448f4cd%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

['Mark S.' via TiddlyWiki]

Seems like the answer might be a separate tag filter operator for use on 
data with high use counts. Then the
developer could decide when it was most appropriate to use whichever tag 
filter operator.


On Sunday, September 22, 2019 at 4:13:40 AM UTC-7, Jed Carty wrote:
>
> Each filter operator only works on the output of the previous operator, so 
> the order matters with later operators in a run generally working with 
> fewer inputs than earlier operators.
> To my knowledge none of the operators do anything in parallel, that may be 
> difficult to set up because javascript is single threaded.
>
> The tags field is treated differently, the tags field is sorted and stored 
> as a different type of list than the title lists normally used in 
> tiddlywiki are, this means that accessing or manipulating the tags field 
> introduces an extra step or two.
>
> In the worst case it has to load the field, parse it as the list type it 
> uses, do whatever the operator does, re-encode it as the tags list and then 
> save it.
>
> Reading or searching the tags field doesn't need to re-encode it or save 
> the field, but the extra parsing can add some overhead.
>
> Other than that each filter step takes a title list and returns a title 
> list, there isn't any other memory that carries over between each step.
>
> The tag operator is a pretty heavy operator. It finds every tiddler in the 
> wiki that has the current tag then checks each input title to see if it is 
> on that list and returns the list of input titles that pass that test.
>
> So if you have a million tiddlers with the tag Vorto it is searching 
> through a list of a million tiddlers once for each input tiddler.
>
> I think that the current version has a better worst-case behaviour than 
> many alternatives, but it's speed is based on the number of tiddlers in the 
> input list and the number of tiddlers in the wiki that have the same tag.
>
> It could be changed so that the speed is based off of the number of input 
> tiddlers and the number of tags each input tiddler has which may be worse 
> in small wikis but a lot better in very large wikis. It depends on the cost 
> of checking the tags list of each tiddler in the input list individually vs 
> the cost of getting a list of all tiddlers in the wiki that have the input 
> tag.
>
> Most of the improvements that immediately come to mind don't preserve the 
> order of the input titles. In large wikis resorting the output could be far 
> faster than the current version, but much slower in smaller wikis.
>
> I don't know how much overhead would be added by adding a check to see 
> which one is faster, that check may be a complex task by itself.
>
> The tag operator may be a place where some significant improvements can be 
> made.
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/6c49003f-e6fe-4139-8c89-49220d4e3cdb%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[Jed Carty]

Each filter operator only works on the output of the previous operator, so 
the order matters with later operators in a run generally working with 
fewer inputs than earlier operators.
To my knowledge none of the operators do anything in parallel, that may be 
difficult to set up because javascript is single threaded.

The tags field is treated differently, the tags field is sorted and stored 
as a different type of list than the title lists normally used in 
tiddlywiki are, this means that accessing or manipulating the tags field 
introduces an extra step or two.

In the worst case it has to load the field, parse it as the list type it 
uses, do whatever the operator does, re-encode it as the tags list and then 
save it.

Reading or searching the tags field doesn't need to re-encode it or save 
the field, but the extra parsing can add some overhead.

Other than that each filter step takes a title list and returns a title 
list, there isn't any other memory that carries over between each step.

The tag operator is a pretty heavy operator. It finds every tiddler in the 
wiki that has the current tag then checks each input title to see if it is 
on that list and returns the list of input titles that pass that test.

So if you have a million tiddlers with the tag Vorto it is searching 
through a list of a million tiddlers once for each input tiddler.

I think that the current version has a better worst-case behaviour than 
many alternatives, but it's speed is based on the number of tiddlers in the 
input list and the number of tiddlers in the wiki that have the same tag.

It could be changed so that the speed is based off of the number of input 
tiddlers and the number of tags each input tiddler has which may be worse 
in small wikis but a lot better in very large wikis. It depends on the cost 
of checking the tags list of each tiddler in the input list individually vs 
the cost of getting a list of all tiddlers in the wiki that have the input 
tag.

Most of the improvements that immediately come to mind don't preserve the 
order of the input titles. In large wikis resorting the output could be far 
faster than the current version, but much slower in smaller wikis.

I don't know how much overhead would be added by adding a check to see 
which one is faster, that check may be a complex task by itself.

The tag operator may be a place where some significant improvements can be 
made.

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/e74c067c-7115-4bb0-baa2-33703b46e97b%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[Mat]

Mark, that might be a huge step forward.

Here is a brain dump of hypotheses and thoughts from someone who doesn't 
really know what he is talking about. You will, for sure, think "what is 
this fool twaddling about?" but while my ignorance is typically a weakness 
it occasionally brings an out-of-the-box perspective that is fruitful. I 
used to think most people were too shy or polite to express ignorant ideas 
but it turned out they just barely have any ideas to begin with - so, for 
what it's worth:


   - I'm assuming that filtering works by saving the step output in a 
   variable that progressively changes for each new step. However if, to our 
   surprise, the position of the tag operator in the filter doesn't affect the 
   time it takes then maybe it happens to not use the latest instance of the 
   variable but an earlier one? A question here is if the variable status is 
   saved somewhere and the tag op somehow accesses this earlier filter step 
   output instead of the latest instance?
   - I believe the tags field is treated differently from other fields. 
   Matching a tag, means iterating over all items in this field. Just maybe 
   the system is made to deal with this in some parallel (asynched?) process 
   and this messes things up when one of the parallel processes needs the 
   input from the other one?
   - Maybe the repeated iteration in the tag operator gets stuck in some 
   recursive loop? Are the edge cases properly investigated - a tiddler with 
   no tag? A tiddler with tag fooX when using tag[foo]? Might some tag fields 
   overflow a magic max number of allowed tags? 
   - And are tiddlers that DO fulfill the filter properly dealt with? For 
   example "tag[foo]" will get the tiddler with "tags: foo bar baz" but will 
   it effectively avoid iterating over the rest of the tags when "foo" is 
   already found? (This possible behaviour would not explain Marks wikis 
   behaviour but maybe it is relevant anyway)
   - Is there some magic number of parallel processes where the *browser* 
   switches methods for performing calculations? Might specifically the tag 
   operator initiate an atypical number of processes which triggers this?
   - Does the tag op - or filters in general - treat pure tags (i.e tags 
   without existing such-named tiddlers) differently from TagTiddlers? 
   - Does the system do any preprocessing on the filter before the steps 
   are processes? Perhaps it cleans up the syntax, e.g trimming white spaces 
   or some such, which affects specifically the tag op?

Again, just a brain dump to trigger ideas. I'm not expecting anyone to 
*answer* these .

<:-)



-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/30626d7b-13c4-4b92-8d4e-fd1bda2b5f4d%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[TonyM]

Mark,

Perhaps try these test in a 5.1.19 Wiki given the additional indexing to 
improve performance was introduced in 5.1.20 and if there was any change 
that could have had an unexpected impact in this area I think it could be 
those changes. Keeping in mind the tags field potentially contains multiple 
values and if different order so how is this sort indexing operating. I 
believe Jeremy's 60K tiddlers were working before 5.1.20 was introduced.

Its great that you are possibly closing in on this. Are you confident your 
sidebar is closed and there is nothing that could be interfering? Like my 
range of suggestions before?

Regards
Tony

On Sunday, September 22, 2019 at 1:35:18 PM UTC+10, Mark S. wrote:
>
> It appears that having a search, list, or list-links that uses a *tag 
> operator*
> running in any currently open tiddler will create a performance problem 
> EVEN if the tag
> operator is applied *AFTER *a previous search operator. 
>
> So this 
>
>
> [all[shadows+tiddlers]search:title{$:/temp/search2}has[en]sort[title]limit[10]]
>
> performs fine, but this
>
>
> [all[shadows+tiddlers]tag[Vorto]search:title{$:/temp/search2}sort[title]limit[10]]
>
> or this
>
>
> [all[shadows+tiddlers]search:title{$:/temp/search2}tag[Vorto]sort[title]limit[10]]
>
> Perform poorly.
>
> This last one is especially puzzling, since you would expect at this point 
> that the tag 
> only has to filter a handful of input titles. This suggests that there 
> might be something 
> wrong with the tag operator, that goes undetected except when there is a 
> large number of tiddlers.
>
> If a process uses the *tag operator* in any open tiddler, then the 
> performance of EVERYTHING,
> including simply typing in some other random tiddler, will be degraded.
>
> Or at least that's what it's looking like to me at the moment.  
>
> Thanks!
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/15b9503c-5251-44b9-a0c3-488dcbccb400%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

['Mark S.' via TiddlyWiki]

It appears that having a search, list, or list-links that uses a *tag 
operator*
running in any currently open tiddler will create a performance problem 
EVEN if the tag
operator is applied *AFTER *a previous search operator. 

So this 

[all[shadows+tiddlers]search:title{$:/temp/search2}has[en]sort[title]limit[10]]

performs fine, but this

[all[shadows+tiddlers]tag[Vorto]search:title{$:/temp/search2}sort[title]limit[10]]

or this

[all[shadows+tiddlers]search:title{$:/temp/search2}tag[Vorto]sort[title]limit[10]]

Perform poorly.

This last one is especially puzzling, since you would expect at this point 
that the tag 
only has to filter a handful of input titles. This suggests that there 
might be something 
wrong with the tag operator, that goes undetected except when there is a 
large number of tiddlers.

If a process uses the *tag operator* in any open tiddler, then the 
performance of EVERYTHING,
including simply typing in some other random tiddler, will be degraded.

Or at least that's what it's looking like to me at the moment.  

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/e4d30087-65ce-42e6-8140-aa5038ba77cb%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

['Mark S.' via TiddlyWiki]

Using "[all[current]has[en]]" did not show any significant improvement.

On Saturday, September 21, 2019 at 12:59:28 PM UTC-7, Mark S. wrote:
>
> Hi Mat,
>
> I've already tried [tag[Vorto]], if that's what you meant 
> by the first part. Alas, it didn't help.
>
> Testing for foofield[Vorto] is a good idea. But I'd have to first modify 
> 36,000 tiddlers. So I'm kind of hoping someone
> will pop up with "Oh, that's a known problem with template tiddlers -- all 
> you have to do is reverse the polarity and you're done" (Reversing
> the polarity always works in ScFi.)
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/63406003-7413-4f33-95c9-9f39c28a91a3%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

['Mark S.' via TiddlyWiki]

Hi Mat,

I've already tried [tag[Vorto]], if that's what you meant 
by the first part. Alas, it didn't help.

Testing for foofield[Vorto] is a good idea. But I'd have to first modify 
36,000 tiddlers. So I'm kind of hoping someone
will pop up with "Oh, that's a known problem with template tiddlers -- all 
you have to do is reverse the polarity and you're done" (Reversing
the polarity always works in ScFi.)

Thanks!

On Saturday, September 21, 2019 at 12:19:42 PM UTC-7, Mat wrote:
>
> Mark S. wrote:
>>
>>
>> <$list filter="[all[current]tag[Vorto]]">
>> Eng: <$edit-text field="en" tag="textarea" rows=2  class="vorto-edit" />
>> 
>>
>
> What happens if you exchange the operators for "synonyms" like instead of 
> "all[current]" do "[[tiddlertitle]tag[Vorto]]" or 
> [all[current]foofield[Vorto]] etc? Maybe it is one of the operators that 
> doesn't function properly.
>
> <:-)
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/c2cbea27-eb51-43a1-8037-2e52cc475bf5%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[Mat]

Mark S. wrote:
>
>
> <$list filter="[all[current]tag[Vorto]]">
> Eng: <$edit-text field="en" tag="textarea" rows=2  class="vorto-edit" />
> 
>

What happens if you exchange the operators for "synonyms" like instead of 
"all[current]" do "[[tiddlertitle]tag[Vorto]]" or 
[all[current]foofield[Vorto]] etc? Maybe it is one of the operators that 
doesn't function properly.

<:-)

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/2768f640-deda-4a69-abfa-66ef3f214b52%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

['Mark S.' via TiddlyWiki]

Hi Birthe, Tony -- everyone! You've all been very helpful!

I think I know *what *is causing the problem -- but I don't know *why*.

There's a view template (WordTemplate) used to expose the en field. The 
contents are typical for templates:

<$list filter="[all[current]tag[Vorto]]">
Eng: <$edit-text field="en" tag="textarea" rows=2  class="vorto-edit" />


Add this to Tony's version (tagged with $:/tags/ViewTemplate) and it will 
immediately bog down
about the same as mine (maybe slightly faster, since the template isn't 
being styled).

This follows the standard recipe for view templates. Tiddlers themselves 
are built from view templates
(or at least the UI we see are built from templates), so why should this 
short bit of code cause 
problems>

I suppose I could jettison the template for this particular project, but 
I'm sure there are other
projects where having a view template for tiddlers will be very important.

Ideas?

Thanks!

Mark


On Thursday, September 19, 2019 at 9:19:37 PM UTC-7, Mark S. wrote:
>
> My project may have been over ambitious. But I thought there were going to 
> be some 
> improvements in 5.1.20+ that would allow for larger TW files. I currently 
> have a TW file
> with about 36,000 short dictionary definitions. Each tiddler consists only 
> of a title
> and a field, EN, with usually only 3 or four words in it. The overall size 
> is only 10 megs.
>
> Overall, the performance is impracticably slow. Ten seconds to open a 
> tiddler. 20 seconds or more
> to do a simple search box search. As much as 25 seconds or more to close 
> out tiddlers that
> need to render a custom search. In most cases, you have to type blind 
> because the
> keystroke refresh is also incredibly slow.
>
> I'm wondering if the performance improvements actually made it into the 
> core, or if
> my expectations were too high. Actually, the performance noticeably 
> declined
> after the first 12,000 entries. 
>
> At the moment it seems like I need to start over, using a data dictionary 
> instead of
> tiddlers for storage. 
>
> Firefox, Windows 7, TW 5.1.21
>
> Thanks!
> Mark
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/d2154326-8d2c-42db-8cb0-e388112a2cc3%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

['Mark S.' via TiddlyWiki]

It's slow even if non-searching tiddlers are all that is rendered.
It's slow with only 6 other tabs open.
It's slow even if FF has just been opened.
It's slow even though instrumentation is off.
It's slow even if doing a single-field match.

It's not just searching that's slow. EVERYTHING you type is slow. You sit 
there waiting ten to 20 seconds for your last
typed entries to appear.


It could be that it's the number of tiddlers rather than the gross size of 
the TW that matters. In which case I suppose 
I could do some crazy design where a single tiddler contains all the words 
that start with "a", etc. But this makes
reverse lookups harder.

Thanks!

On Thursday, September 19, 2019 at 10:28:26 PM UTC-7, TonyM wrote:
>
> Mark,
>
> To fully address your experience consider if you can show or share the 
> wiki. Alternativy share some test data of an equivalent size. Also consider 
> it from a "what must it render perspective". If for example the tiddlers 
> were all listed in the table of contents and this is displayed in the 
> sidebar this will refresh every change that may effect that toc list. Hide 
> the side bar and see if its the same.
>
> I had 12Mb files working well in TiddlyWiki Classic, so it should be good 
> in TW5
>
> In an App I built there was a considerable difference with 5.1.20 however 
> be warned there was an accidental leaving on of "Performance 
> instrumentation", see Control Panel > Settings > Performance 
> instrumentation - uncheck save and reload.
>
> Also a normal search that is activated for any three letter combination 
> and lists the result below may just be inappropriate for such an 
> application. Perhaps you need a enter search criteria then hit a search 
> button, which renders the results in a list, rather than an instant 
> response popup/list. 
>
> There is also a little gotcha with browser memory. Try with nothing else 
> open by or in a tab of the browser and see if it improves, this will give 
> us a clue to its bottle necks. Browsers also have a maximum memory usage 
> set so they do not overpower the device/computer but this is often a fixed 
> value, when you mostly work in the browser there is value making more ram 
> available to the browser especially if you have 8-16GB available.
>
> The more you share the more we can help.
>
> Regards
> Tony
> On Friday, September 20, 2019 at 2:19:37 PM UTC+10, Mark S. wrote:
>>
>> My project may have been over ambitious. But I thought there were going 
>> to be some 
>> improvements in 5.1.20+ that would allow for larger TW files. I currently 
>> have a TW file
>> with about 36,000 short dictionary definitions. Each tiddler consists 
>> only of a title
>> and a field, EN, with usually only 3 or four words in it. The overall 
>> size is only 10 megs.
>>
>> Overall, the performance is impracticably slow. Ten seconds to open a 
>> tiddler. 20 seconds or more
>> to do a simple search box search. As much as 25 seconds or more to close 
>> out tiddlers that
>> need to render a custom search. In most cases, you have to type blind 
>> because the
>> keystroke refresh is also incredibly slow.
>>
>> I'm wondering if the performance improvements actually made it into the 
>> core, or if
>> my expectations were too high. Actually, the performance noticeably 
>> declined
>> after the first 12,000 entries. 
>>
>> At the moment it seems like I need to start over, using a data dictionary 
>> instead of
>> tiddlers for storage. 
>>
>> Firefox, Windows 7, TW 5.1.21
>>
>> Thanks!
>> Mark
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/2ef5aae4-83c9-411e-bb62-ad97832f6b2c%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[Birthe C]

Hi Mark S,
You are around the size 10 megs where I usually get into trouble. Recent i 
got a "new" computer with more ram, it did help a little but really not 
enough for it to be fun to work with these wikies..

Former I sometimes used Firefox, Windows7 and found it slower than Firefox, 
Linux Mint.


Birthe

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/e79d51ae-c8f7-4d72-8a44-74d4ca21a5fb%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[@TiddlyTweeter]

Mark S. wrote:
>
>  At the moment it seems like I need to start over, using a data 
> dictionary instead of
> tiddlers for storage. 
>

I'd be very interested to know, if you go ahead with a data dictionary 
version, what the performance is like!

Best wishes
TT

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/345b763b-3623-4052-ba71-1ba789f0f601%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[Mohammad]

TW-Scripts has 4000+ tiddlers including all shadow tiddlers.
I am on 5.1.21, Firefox 69, Win 10

I see noticeable response time comparing to empty.html

--Mohammad
 

On Friday, September 20, 2019 at 8:49:37 AM UTC+4:30, Mark S. wrote:
>
> My project may have been over ambitious. But I thought there were going to 
> be some 
> improvements in 5.1.20+ that would allow for larger TW files. I currently 
> have a TW file
> with about 36,000 short dictionary definitions. Each tiddler consists only 
> of a title
> and a field, EN, with usually only 3 or four words in it. The overall size 
> is only 10 megs.
>
> Overall, the performance is impracticably slow. Ten seconds to open a 
> tiddler. 20 seconds or more
> to do a simple search box search. As much as 25 seconds or more to close 
> out tiddlers that
> need to render a custom search. In most cases, you have to type blind 
> because the
> keystroke refresh is also incredibly slow.
>
> I'm wondering if the performance improvements actually made it into the 
> core, or if
> my expectations were too high. Actually, the performance noticeably 
> declined
> after the first 12,000 entries. 
>
> At the moment it seems like I need to start over, using a data dictionary 
> instead of
> tiddlers for storage. 
>
> Firefox, Windows 7, TW 5.1.21
>
> Thanks!
> Mark
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/89274b35-487f-47d4-b953-af494a66ad4d%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[TonyM]

Post Script

If you a doing word searches, ie words as tiddler titles, not words in 
phrases or in the test fields changing the search not to look only in 
titles (not in text) or even search first by prefix could improve search 
outcomes a lot.

Regards
Tony

On Friday, September 20, 2019 at 3:28:26 PM UTC+10, TonyM wrote:
>
> Mark,
>
> To fully address your experience consider if you can show or share the 
> wiki. Alternativy share some test data of an equivalent size. Also consider 
> it from a "what must it render perspective". If for example the tiddlers 
> were all listed in the table of contents and this is displayed in the 
> sidebar this will refresh every change that may effect that toc list. Hide 
> the side bar and see if its the same.
>
> I had 12Mb files working well in TiddlyWiki Classic, so it should be good 
> in TW5
>
> In an App I built there was a considerable difference with 5.1.20 however 
> be warned there was an accidental leaving on of "Performance 
> instrumentation", see Control Panel > Settings > Performance 
> instrumentation - uncheck save and reload.
>
> Also a normal search that is activated for any three letter combination 
> and lists the result below may just be inappropriate for such an 
> application. Perhaps you need a enter search criteria then hit a search 
> button, which renders the results in a list, rather than an instant 
> response popup/list. 
>
> There is also a little gotcha with browser memory. Try with nothing else 
> open by or in a tab of the browser and see if it improves, this will give 
> us a clue to its bottle necks. Browsers also have a maximum memory usage 
> set so they do not overpower the device/computer but this is often a fixed 
> value, when you mostly work in the browser there is value making more ram 
> available to the browser especially if you have 8-16GB available.
>
> The more you share the more we can help.
>
> Regards
> Tony
> On Friday, September 20, 2019 at 2:19:37 PM UTC+10, Mark S. wrote:
>>
>> My project may have been over ambitious. But I thought there were going 
>> to be some 
>> improvements in 5.1.20+ that would allow for larger TW files. I currently 
>> have a TW file
>> with about 36,000 short dictionary definitions. Each tiddler consists 
>> only of a title
>> and a field, EN, with usually only 3 or four words in it. The overall 
>> size is only 10 megs.
>>
>> Overall, the performance is impracticably slow. Ten seconds to open a 
>> tiddler. 20 seconds or more
>> to do a simple search box search. As much as 25 seconds or more to close 
>> out tiddlers that
>> need to render a custom search. In most cases, you have to type blind 
>> because the
>> keystroke refresh is also incredibly slow.
>>
>> I'm wondering if the performance improvements actually made it into the 
>> core, or if
>> my expectations were too high. Actually, the performance noticeably 
>> declined
>> after the first 12,000 entries. 
>>
>> At the moment it seems like I need to start over, using a data dictionary 
>> instead of
>> tiddlers for storage. 
>>
>> Firefox, Windows 7, TW 5.1.21
>>
>> Thanks!
>> Mark
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/f6466c28-7259-4d1a-8fd2-8248d9d2c800%40googlegroups.com.

[tw5] Re: Hitting a barrier at 36,000 tiddlers

[TonyM]

Mark,

To fully address your experience consider if you can show or share the 
wiki. Alternativy share some test data of an equivalent size. Also consider 
it from a "what must it render perspective". If for example the tiddlers 
were all listed in the table of contents and this is displayed in the 
sidebar this will refresh every change that may effect that toc list. Hide 
the side bar and see if its the same.

I had 12Mb files working well in TiddlyWiki Classic, so it should be good 
in TW5

In an App I built there was a considerable difference with 5.1.20 however 
be warned there was an accidental leaving on of "Performance 
instrumentation", see Control Panel > Settings > Performance 
instrumentation - uncheck save and reload.

Also a normal search that is activated for any three letter combination and 
lists the result below may just be inappropriate for such an application. 
Perhaps you need a enter search criteria then hit a search button, which 
renders the results in a list, rather than an instant response popup/list. 

There is also a little gotcha with browser memory. Try with nothing else 
open by or in a tab of the browser and see if it improves, this will give 
us a clue to its bottle necks. Browsers also have a maximum memory usage 
set so they do not overpower the device/computer but this is often a fixed 
value, when you mostly work in the browser there is value making more ram 
available to the browser especially if you have 8-16GB available.

The more you share the more we can help.

Regards
Tony
On Friday, September 20, 2019 at 2:19:37 PM UTC+10, Mark S. wrote:
>
> My project may have been over ambitious. But I thought there were going to 
> be some 
> improvements in 5.1.20+ that would allow for larger TW files. I currently 
> have a TW file
> with about 36,000 short dictionary definitions. Each tiddler consists only 
> of a title
> and a field, EN, with usually only 3 or four words in it. The overall size 
> is only 10 megs.
>
> Overall, the performance is impracticably slow. Ten seconds to open a 
> tiddler. 20 seconds or more
> to do a simple search box search. As much as 25 seconds or more to close 
> out tiddlers that
> need to render a custom search. In most cases, you have to type blind 
> because the
> keystroke refresh is also incredibly slow.
>
> I'm wondering if the performance improvements actually made it into the 
> core, or if
> my expectations were too high. Actually, the performance noticeably 
> declined
> after the first 12,000 entries. 
>
> At the moment it seems like I need to start over, using a data dictionary 
> instead of
> tiddlers for storage. 
>
> Firefox, Windows 7, TW 5.1.21
>
> Thanks!
> Mark
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/1b52c64f-6798-4799-b2a9-b839fb5afe4f%40googlegroups.com.