Hi h0p3 Great post, thank you for sharing your hopes and fears!
With regard to performance improvements and optimisations in general, I’ve found that whenever I’ve gone in pursuit of them I’ve been able to squeeze better and better performance out of the core itself. As you know, I’m working with very large wikis too, and as I’ve encountered serious issues I’ve tried to fix them, and largely succeeded. I’ve found that to be true at all levels of the design: if I can measure the performance of something then I can almost certainly improve it. The vital component is to be working with real-world data. With regard to browsers, I’m much more bullish that we’ll continue to see significant improvements, particularly for the large processes that browsers require for these large wikis. One interesting point about TiddlyWiki is that the refresh algorithm is inherently parallelisable: refresh processing of sibling nodes is completely independent of one another. So, to the extent that browsers will expose more multicore functionality in their quest for performance, we’re in a good position to benefit. Looking further to the future, my thinking for some time around TWX is to write it in an LLVM-compatible language, and compile it to WebAssembly for the Web. (I hesitated to write that because we really shouldn’t de-rail this discussion, but it does seem relevant that I’m planning a long term future for TiddlyWiki). Anyhow, probably the best we can do at the moment is to continue to improve the performance monitoring tools so that anyone can make observations of their own wikis, and we can figure out how best to act on the resulting telemetry (i.e. without everybody having to send a copy of their private wikis to me!) Best wishes Jeremy > On 6 Jul 2019, at 04:00, h0p3 <[email protected]> wrote: > > What does the future hold for large Tiddlywikis? What can I do right now to > start optimizing my TW to be usable while still huge. I am grateful to the TW > contributors and those who made one my browser engines. This has to be one of > the best FOSS communities I've ever had the privilege of participating in. I > need the advice of experts. > > There's only so much optimization TW5 and the browsers which run it can > achieve. We're hitting the limits of single-threaded performance, and I am > not convinced there are many huge gains left for javascript in the browser on > high-end x86_64 CPUs (I desperately hope I'm wrong). Performance for large > TWs is looking a bit grim. I'm trying to look down the road 10 years from now > and ask myself if my Tiddlywiki is going to be the right tool for the job. > It's an amazing rapid-prototyping tool, but maybe it can't be made performant > enough. What do you think? > > Even though I'm still a noob at it, I use TW a lot > <https://philosopher.life/>; I'm an addict. In 3 years, I've amassed 10k > tiddlers of almost pure text in a 30MB TW (that's ~60 novels in length), and > I don't see myself slowing down. I don't do anything fancy, but this tool is > heavily integrated into my life. I'm a unificationist too: part of the > strength of this tool is that I don't have to separate it into unconnected > documents. I want to search, navigate, hyperlink, and construct the whole. > Unfortunately, the tool is getting slower and slower for me. I'm pretty > worried I need to move away from Tiddlywiki. > > The standout property of Tiddlywiki is that I get to serve a self-modifying > IDE+Product as a single html file (though I no longer develop it without > Bob). The client's browser does most of the calculation, and I don't have to > rely upon having a server which does anything more than dishing out static > files; I'm not beholden to centralized webservers (though I still use github > for now). I adore how it is censorship resistant, cryptographically signable, > easy to distribute, and it runs on almost any device (though, at this point, > my wiki barely runs on a phone). It's perfect for P2P-serverlessness. There > is nothing else like it in this respect unless I'm handing someone a complete > VM or container, but that doesn't work nicely in a browser. There is no > replacement as far as I can tell. > > I've probably made plenty of mistakes in attempting to optimize, but I'm > trying. I try to stick to hardlinks, and I do my best not to generate > anything dynamically when I can. There's only so much optimizing one can do > for complex filter expressions. I even bend over backward to do what I call > "Firmcoding" in which dynamically generated lists are pre-computed into > static indexes (basically, this enables link references to function, limits > clientside computation, and if I have to move out of Tiddlywiki, all of my > linking structures are still hardcoded). What kinds of precomputing can I do > here to help the client side? I've only started really using tags this past > year, and I just ripped out all of the tags to find almost no performance > gains either. Do I have too many tiddlers? Do I need to start finding ways to > molecularize (rather than atomize) content into larger tiddler bodies and > then build specialized parsing for those large tiddlers? I'm growing > desperate. > > I feel like I'm going to puke. I've seen this coming for a while, and I've > been burying my head in the sand hoping it wasn't true. This is my evolving > horcrux-pensieve, and now I feel like a hermit crab who might have to find > another shell. I'm heartbroken at the thought. This is the best damn tool I > have ever used in my entire life; I can't bear to lose it. NO! NO! > NOO!!!!!!!!! /hissy-fit. > > Is there no hope for me here? Will the next TW-X be built with WASM? What > other toolset do you recommend for me? Where should this pilgrim go, and what > should he do? > > -- > You received this message because you are subscribed to the Google Groups > "TiddlyWiki" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/tiddlywiki > <https://groups.google.com/group/tiddlywiki>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/tiddlywiki/ef383bab-eed0-4549-b407-b16dba96dae9%40googlegroups.com > > <https://groups.google.com/d/msgid/tiddlywiki/ef383bab-eed0-4549-b407-b16dba96dae9%40googlegroups.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "TiddlyWiki" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/tiddlywiki. To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/4016F832-7E90-44CF-B929-76A8C4088632%40gmail.com. For more options, visit https://groups.google.com/d/optout.
