Hey, Jeremy — With what you've told me about the preboot approach on the call, it's starting to sound a bit better for modloader's needs. Learning that the serialization DOM format is fixed across all TW5 platforms helps me to visualize a solution.
I'm wondering if this pre-boot approach will be incompatible with encrypted wikis, though. I have at least one of those that I'd really like to use patches with, if possible. Are plugins (including core) encrypted, or are they excluded from that mechanism? * The proposed modifications don’t have any anticipated benefit beyond the > usage by the modloader plugin I will continue to voice my dissent to this argument. There are other useful mechanisms that might want to add code before startup. For example, a plugin could fetch some widget script via HTTPS in order to ensure that client wikis are always using the latest version. Arguably this could be done in pre-boot like the modloader, too, but it's an example of something that needs to deal with similar constraints. On Tuesday, 13 February 2018 05:57:07 UTC-6, Jeremy Ruston wrote: > > Hi Evan > > The other problem is that my plugin's requirement — "minimize the number > of modules executing before me" — is at odds with the dependency-oriented > nature of startup modules. Modloader doesn't depend on any functionality > other than what's provided by the bootloader. > > > “Minimize the number of modules executing before me” is the key phrase > that triggers my concern about an arms race; I’ve seen it so many times > before from CSS to TSRs, and introducing another category is never a good > long term answer. The pre-boot approach is attractive precisely because > it’s outside the normal mechanisms of TiddlyWiki, and hence can be used to > patch/hack those mechanisms. It is there to help avoid the kind of > patching-the-rug-you’re-standing-on situation we face here. > > Aren’t (the executions before startup) just the cascade of modules >> requiring one another? >> > > Yes. The requirement for my plugin is "run before most modules execute". > There's no best practice for startup modules to discourage them from > loading modules at execution time, and I'm not sure there should be(?), > hence the conundrum. > > You'll notice my modloader plugin eclipses startup.js to move its > require() call into its startup() method. If I don't do this, it's > impossible to modify a large portion of the core (including all widgets). > Unfortunately, this eclipsing is exactly the kind of thing the modloader is > designed to avoid. > > > I’ve no objection to moving existing require() calls into a different > scope. > > It might be a useful exercise to try to write the documentation for >> bootloader modules, and see whether one can explain the difference clearly >> without referencing the particular use case that has prompted this >> discussion. >> > > "bootloader modules run during the boot process, allowing them to add, > remove and modify tiddlers in the store prior to the wiki's startup > process. This is useful for injecting plugins, code and content that > TiddlyWiki isn't capable of refreshing at runtime. Injected content may be > dynamically generated or loaded from an external source. These modules are > encouraged to minimize their dependency upon other modules in the wiki and > to avoid any dependency on the TiddlyWiki core. > > bootloader modules may safely revise any content in the tiddler store with > the exception of other bootloader modules and their dependencies.” > > > And that sounds pretty much like a description of the existing > $tw.preloadTiddlers mechanism. The preboot approach offers the interesting > possibility of patching boot.js, too. > > I’m happy to discuss further but my position remains: > > * The proposed modifications don’t have any anticipated benefit beyond the > usage by the modloader plugin > * The proposed modifications are relatively expensive in terms of time and > attention from me and the rest of the core team > * The modloader plugin already exists and works, with some limitations > * The preboot approach already exists as an alternative mechanism that > could be used instead, with different limitations > > Best wishes > > Jeremy > > > > > On Monday, 12 February 2018 16:53:53 UTC-6, Jeremy Ruston wrote: >> >> Hi Evan >> >> Thanks, I’ll give your arguments more thought, and hope you can join the >> hangout tomorrow to discuss further. >> >> This is almost, but not quite, the issue. My modloader *is* the first >> module to have any method called on it. However, TiddlyWiki currently >> *loads* (via $tw.module.execute) a very large number of modules prior to >> making any method calls. There is no control over the ordering of these >> executions, and they are subsequently impossible to change at boot-time >> without overwriting the core. >> >> >> Aren’t those executions just the cascade of modules requiring one another? >> >> As I mentioned before, I think there's potentially a meaningful >> difference between startup modules and "bootloader" behaviors. Loading >> content and code into the tiddler store, versus setting up APIs, DOMs and >> run-time functionality. It's much more likely that a loader module would >> want to affect startup behavior than another loader module. >> >> >> Bootloader modules feel a bit like an admission of defeat with respect to >> the startup module mechanism. >> >> It might be a useful exercise to try to write the documentation for >> bootloader modules, and see whether one can explain the difference clearly >> without referencing the particular use case that has prompted this >> discussion. >> >> I worry that they are a narrow solution to your specific problem. We >> could also be looking at something buggyj has already started: making it >> easier for plugins to splice in raw markup code, and possibly adding more >> hooks to the boot kernel (eg, an explicit “patch-tiddler” hook). >> >> Anyway, the modloader doesn't *need* this functionality, strictly >> speaking. It just extends its capabilities a bit, so that the *whole* core >> can be patched by installed plugins. The modloader currently issues a >> warning when trying (and failing) to apply a patch of this nature so it >> shouldn't catch developers by surprise. >> >> >> Excellent. >> >> The design intent of the modloader is to maintain compatibility with >> future versions of TiddlyWiki. Duplicating bootloader behavior (which is >> subject to change) would detrimental enough to that goal that I'd rather >> stick with the current constraints. >> >> >> But I think the code that you’d need to duplicate is code that’s only >> going to change if the underlying serialisation formats change? If that >> happens all bets are off with respect to future versions of TiddlyWiki >> maintaining backwards compatibility with this February 2018 incarnation of >> the plugin. >> >> Best wishes >> >> Jeremy. >> >> >> >> On Monday, 12 February 2018 15:10:16 UTC-6, Jeremy Ruston wrote: >>> >>> Hi Evan >>> >>> We’ve got to the point where the changes you are proposing to the core >>> are quite deep, and have lots of implications for the rest of the core. >>> It’s going to mean a lot of work on verification and testing from me, even >>> if somebody else writes the code. I don’t think that burden is justified at >>> present. >>> >>> Better to work with the plugin for a while, and explore the capabilities >>> and consequences before we commit to a new mechanism in the core. >>> >>> So, as mentioned in the OP I'm pretty leery of the "pre-boot" approach >>> here. The modloader process >>> <http://evanbalster.com/tiddlywiki/formulas.html#%24%3A%2Fplugins%2Febalster%2Fmodloader%2Floader.js> >>> uses >>> quite a lot of $:/boot library functionality to identify and inspect shadow >>> tiddlers as part of the patching process, and to generate the output plugin. >>> >>> >>> At this point, I don’t see any problem with your plugin duplicating a >>> few hundred lines of boot.js; eliminating such duplication would come >>> firmly under the heading of optimisation. >>> >>> I also worry that working on the pre-boot DOM would create >>> incompatibilities with certain TiddlyWiki environments (but I don't >>> understand this low level stuff very well). >>> >>> >>> You can see all the code that depends on this stuff. The approach is >>> solid. >>> >>> The current modloader can manipulate almost everything in the wiki >>> *except *for startup modules and the modules they require, hence the >>> request. >>> >>> >>> Yes, but your proposed solution is just an escalation in an ongoing arms >>> race to be the first module to run. >>> >>> In most cases, bootloader modules (including modloader) are going to >>> want to generate any output to $:/temp or similar, where they won't create >>> persistent changes in the wiki. >>> >>> >>> But equally, there’s no limit to the functionality in the boot kernel >>> that experimental plugins might want to tweak. Better to be based on a >>> pre-boot hook, and have the capability to hook anything, than to be >>> dependent on the hooks anticipated and provided by the core. >>> >>> Best wishes >>> >>> Jeremy. >>> >>> So, as mentioned in the OP I'm pretty leery of the "pre-boot" approach >>> here. The modloader process >>> <http://evanbalster.com/tiddlywiki/formulas.html#%24%3A%2Fplugins%2Febalster%2Fmodloader%2Floader.js> >>> uses >>> quite a lot of $:/boot library functionality to identify and inspect shadow >>> tiddlers as part of the patching process, and to generate the output >>> plugin. I also worry that working on the pre-boot DOM would create >>> incompatibilities with certain TiddlyWiki environments (but I don't >>> understand this low level stuff very well). The current modloader can >>> manipulate almost everything in the wiki *except *for startup modules >>> and the modules they require, hence the request. >>> >>> In most cases, bootloader modules (including modloader) are going to >>> want to generate any output to $:/temp or similar, where they won't create >>> persistent changes in the wiki. >>> >>> I expect we'll end up discussing in more depth during tomorrow's call. >>> >>> On Monday, 12 February 2018 14:09:17 UTC-6, Jeremy Ruston wrote: >>>> >>>> Here’s the .tid of the $:/PatchCore.js tiddler, too. >>>> >>>> Best wishes >>>> >>>> Jeremy >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "TiddlyWikiDev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/tiddlywikidev. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/tiddlywikidev/98a6ce14-9ba5-470d-b396-365f08ddd806%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/tiddlywikidev/98a6ce14-9ba5-470d-b396-365f08ddd806%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "TiddlyWikiDev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to tiddly...@googlegroups. >> <http://googlegroups.com/>com <http://googlegroups.com/>. >> Visit this group at https://groups.google.com/group/tiddlywikidev. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/tiddlywikidev/cf2e1886-225d-49a0-9232-c646ee5352f7%40googlegroups.com >> >> <https://groups.google.com/d/msgid/tiddlywikidev/cf2e1886-225d-49a0-9232-c646ee5352f7%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "TiddlyWikiDev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] > <javascript:>. > Visit this group at https://groups.google.com/group/tiddlywikidev. > To view this discussion on the web visit > https://groups.google.com/d/msgid/tiddlywikidev/9a8c3a1b-e53c-42b1-9cde-c3a4bbcc7759%40googlegroups.com > > <https://groups.google.com/d/msgid/tiddlywikidev/9a8c3a1b-e53c-42b1-9cde-c3a4bbcc7759%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/tiddlywikidev. To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/9e519051-0aca-44c1-b2d1-9f71c01eb4bb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
