Re: [tw] Re: How to handle a lot of wikis with Node

[Tristan Kohl]

I do not have much time right now unfortunately but as I see from the 
Dev-Tools there is only one initial load of the wiki. Afterwards it just 
periodically polls the Tiddler list. Since you know more about the ins and 
outs of TW would it be hard to just let TW generate this initial view for 
multiple wikis determined by the URL and consecutively the tiddlers 
JSON? As my 99% text-only tiddlers do not eat up that much memory I would 
be fine if they were all stored within Node. I only need to get rid of 
redundant Node instances all loading a full TW environment and snacking 
away my limited RAM 30 - 40 MB at a time.

On Thursday, February 15, 2018 at 4:54:19 AM UTC+1, Arlen Beiler wrote:
>
> You have precisely defined the problem. The crux of the matter really is 
> the cache because it isn't just a cache, it is the entire $tw environment 
> loaded into NodeJS, which then generates the HTML for the browser. 
>  
>
>> I think one has to fix the server command as well as it defaults to the 
>> "default" route and just needs to use the value from bag to determine the 
>> correct target wiki.
>>
>> Other than that I did not find any problems that surfaced on my first 
>> look. It would be fine for me if the plugins are just forwarded to the 
>> browser as all the real work happens there.
>>
>> Side note: I thought about writing a SQLite adaptor some time if my setup 
>> grows as I can imagine this would speed up things quite a lot and make 
>> backups easier since all wikis would be just tables within one database. 
>> The only problem I saw just yet is the fact that the Node server seems to 
>> cache the tiddlers - otherwise mult-user wikis would be a breeze as Node 
>> could be completely agnostic of anything and just serve the tiddlers from 
>> the database. But I think this is a topic for a different day.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/b71b135c-1f77-4855-be22-518bad82a5a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: How to handle a lot of wikis with Node

[Arlen Beiler]

You have precisely defined the problem. The crux of the matter really is
the cache because it isn't just a cache, it is the entire $tw environment
loaded into NodeJS, which then generates the HTML for the browser.


> I think one has to fix the server command as well as it defaults to the
> "default" route and just needs to use the value from bag to determine the
> correct target wiki.
>
> Other than that I did not find any problems that surfaced on my first
> look. It would be fine for me if the plugins are just forwarded to the
> browser as all the real work happens there.
>
> Side note: I thought about writing a SQLite adaptor some time if my setup
> grows as I can imagine this would speed up things quite a lot and make
> backups easier since all wikis would be just tables within one database.
> The only problem I saw just yet is the fact that the Node server seems to
> cache the tiddlers - otherwise mult-user wikis would be a breeze as Node
> could be completely agnostic of anything and just serve the tiddlers from
> the database. But I think this is a topic for a different day.
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/CAJ1vdSSqsW3aoc2-ORfrMe3SFk9RLr6ksy0caxNe7hta%3D-d6Tw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: How to handle a lot of wikis with Node

[Tristan Kohl]

Hi Arlen,

nice to hear from you again (thanks for fixing my image linking problems a 
while back over on github :) ).

Since there was no answer that could help me up until you came I finally 
had a good reason to overcome my own weaker self and start to dig into TW 
code for real. As I am quite confident in JS but was just to lazy to ever 
bother before I quickly came to the conclusion that the changes I would 
have to implement are less dramatic than I expected. As the tiddlyweb 
adaptor already uses the "bag" field but only fills it with "default", 
changing this behavior to make the field contain the actual wiki name 
should be an easy task. I just started setting up a dev environment the 
other day but was not able to write any code just yet.

I think one has to fix the server command as well as it defaults to the 
"default" route and just needs to use the value from bag to determine the 
correct target wiki.

Other than that I did not find any problems that surfaced on my first look. 
It would be fine for me if the plugins are just forwarded to the browser as 
all the real work happens there.

Side note: I thought about writing a SQLite adaptor some time if my setup 
grows as I can imagine this would speed up things quite a lot and make 
backups easier since all wikis would be just tables within one database. 
The only problem I saw just yet is the fact that the Node server seems to 
cache the tiddlers - otherwise mult-user wikis would be a breeze as Node 
could be completely agnostic of anything and just serve the tiddlers from 
the database. But I think this is a topic for a different day.

If you need any help in implementing the changes, I would be willing to 
throw in a hand.

Cheers,
Tristan

On Tuesday, February 13, 2018 at 4:38:03 PM UTC+1, Arlen Beiler wrote:
>
> I came across this thread this evening. I like hearing that someone else 
> besides me has grappled with this problem. You in using TiddlyServer, and I 
> in developing it. 
>
> This is exactly the problem that I predicted would happen, but I did not 
> expect it would show up in real-world use, as I did not really think anyone 
> would have 50 wikis! I should have known to trust my instincts :)
>
> The problem is caused in part by the way Tiddlywiki data folders are 
> setup. Each folder must be in its own Tiddlywiki environment. So every wiki 
> loads the same code files from disk and executes them. All in the same Node 
> process, but still all separate objects.
>
> I could create a new type of connecter that would serve the folder 
> directly to the client along with the required plugins and core. When a 
> data folder would be opened it would serve a loader which would load the 
> core and the specified plugins directly into the browser and then sync 
> changes back to the server. 
>
> In my mind this should be a fairly easy route to follow. The loader could 
> be generated on the fly so the page loads fairly quickly and also so it 
> works with the browser cache. I have wanted to implement this in many 
> scenarios and it seems like it is becoming more realistic and necessary.
>
> Thoughts, anyone? Does it matter if plugins in a data folder get loaded 
> directly to the browser? It shouldn't, I don't think. Especially with the 
> current template for the server command in TiddlyServer, which is 
> $:/core/save/all.
>
> That's my two cents.
> Arlen
>
> On Feb 12, 2018 00:46, "Tristan Kohl"  
> wrote:
>
>> Unfortunetely not, nohup just stops them from receiving the HUP signal 
>> when I terminate a ssh session. Nothing in terms of overhead due to loading 
>> a new Node instance running full TW for every wiki.
>>
>> On Sunday, February 11, 2018 at 1:54:37 PM UTC+1, TonyM wrote:
>>>
>>> Only a short response but on node instance with starting multiple wikis 
>>> with nohup  & does that reduce overheads?
>>>
>>> Tony
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "TiddlyWiki" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to tiddlywiki+...@googlegroups.com .
>> To post to this group, send email to tiddl...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/tiddlywiki.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/tiddlywiki/8f082316-2de6-4838-b6fc-1907d61a2935%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this 

Re: [tw] Re: How to handle a lot of wikis with Node

[Arlen Beiler]

I came across this thread this evening. I like hearing that someone else
besides me has grappled with this problem. You in using TiddlyServer, and I
in developing it.

This is exactly the problem that I predicted would happen, but I did not
expect it would show up in real-world use, as I did not really think anyone
would have 50 wikis! I should have known to trust my instincts :)

The problem is caused in part by the way Tiddlywiki data folders are setup.
Each folder must be in its own Tiddlywiki environment. So every wiki loads
the same code files from disk and executes them. All in the same Node
process, but still all separate objects.

I could create a new type of connecter that would serve the folder directly
to the client along with the required plugins and core. When a data folder
would be opened it would serve a loader which would load the core and the
specified plugins directly into the browser and then sync changes back to
the server.

In my mind this should be a fairly easy route to follow. The loader could
be generated on the fly so the page loads fairly quickly and also so it
works with the browser cache. I have wanted to implement this in many
scenarios and it seems like it is becoming more realistic and necessary.

Thoughts, anyone? Does it matter if plugins in a data folder get loaded
directly to the browser? It shouldn't, I don't think. Especially with the
current template for the server command in TiddlyServer, which is
$:/core/save/all.

That's my two cents.
Arlen

On Feb 12, 2018 00:46, "Tristan Kohl"  wrote:

> Unfortunetely not, nohup just stops them from receiving the HUP signal
> when I terminate a ssh session. Nothing in terms of overhead due to loading
> a new Node instance running full TW for every wiki.
>
> On Sunday, February 11, 2018 at 1:54:37 PM UTC+1, TonyM wrote:
>>
>> Only a short response but on node instance with starting multiple wikis
>> with nohup  & does that reduce overheads?
>>
>> Tony
>>
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tiddlywiki+unsubscr...@googlegroups.com.
> To post to this group, send email to tiddlywiki@googlegroups.com.
> Visit this group at https://groups.google.com/group/tiddlywiki.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/tiddlywiki/8f082316-2de6-4838-b6fc-1907d61a2935%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/CAJ1vdSSft7w%3DWp5CBAXm-zHogpA5kUxeP6Y6Uib4WYZB0sTSQA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.