Hi Jeremy, thanks for stepping in, > Thus, it seems to me that we really need 2 things: > > - nice docs describing how to support saving of TWC, TW5 (and, in > perpective, more features like image uploading and more) > > > - core libraries in different languages (js for browser plugins? js > for node, PHP, Java/Android, Swift/iOS, may be more) which one can use for > developing new solutions > > Do you mean docs for writing an app for handling saving of TiddlyWikis? Or > for building a serverside for TiddlyWikis? Or do you mean docs for how the > saving mechanisms work? >
Well, I mean docs for (writing an app/backend for) handling saving in the first place. But later those can cover supporting more features and I don't really see the boundary where one can call those "docs for building serverside" (a primitive serverside *is* a script which just saves TW). Of'course the docs may later cover topics like authentication, or uploading images etc, but the MVP is "how to support saving TWs". Another point about such docs/repo is those can be a more structured way to discuss and understand current and future architecture of TWC/5, so one can think of them also as [infrastructure for] a process, not a product. (for instance, I never really understood what adaptors are exactly) That said, also think that saving engine of TWC should be updated as well and I'm even considering refactor it for easier support of async and granulated saving (already supported by MTS, see implementOnlineSaving <https://github.com/YakovL/MainTiddlyServer/blob/master/index.php#L292> in the injected JS), may be by making it similar to TW5. By the way, I'm a bit confused with TF's handling of TW5 saving (or rather the saving mechanism itself) in a couple of aspects, but let me go into details below. I think we are indeed missing docs for the .tid file format, but probably > only because it’s so basic. Are you running into a problem with it? > Well, I have at least one question about it. As far as I understand, it looks like: field: value field2: value2 title: some tiddler tiddler text goes here, after 2 linebreaks (watch out for CRLF!) so is there support for multiline fields? I have a bunch of tiddlers in my TWs which have multiline field values, so it's not purely theoretical (I believe using fields for database-like approach is generally somewhat natural as well as multiline content in those). Now to the TW5 and TF aspects. Like I said, I understand that the "tiddlywiki-save-file" custom event (with .path and .content properties) is dispatched when saving is needed. What I don't understand is how TW5 is gets back the confirmation of changes being saved (and which set of changes was saved), because there can be "disconnections" – in case of TF it can be ejected USB-storage with TW, in case of a server – absence of network connection. I also wonder if the current architecture of granulated saving in TW5 is compatible with undoing changes (is there a saved queue of changes?). This is not of major importance, but it's better to think about it to avoid another refactoring in the future. As for TWC, TF actually introduces a bug: since the overwritten mozillaSaveFile always returns true, with newer (2.8.1+) versions saveMain reports changes being saved even if USB-storage with TW was ejected (it wasn't the case in earlier versions of TWC because loadOriginal always loaded original from file system, and saveMain stopped before calling mozillaSaveFile if TW is not currently available). > I'll start with TiddlyFox since it is still used by many users (with > either FireFox 56- or Pale Moon, WaterFox and other FF forks). One thing > about it is building the xpi: I've read about building it a little and the > crucial quesion is: is it still possible to create signed xpi, or updating > TiddlyFox is impossible in principle? > > Mozilla was responsible for signing the extension as part of the process > of uploading it; I don’t think there’s ever been a way for anyone else to > sign it and have it accepted by recent versions of Firefox. But I don’t > know much about it, it may be worth asking on the Mozilla dev mailing lists. > I'll be thankful if you suggest one [mailing list] to begin with. I can explore the topic myself, but since you have some experience, may be it's easier for you to point where to look. (hmm.. I wonder if Pale Moon requires same singing as FF or they have built another infrastructure, may be worth researching) No problem with adding you to the repo. > Right, would be nice. Ok, let me discuss only a fraction of technical details in this reply to answer sooner and to focus on organization aspects first (and actually technical details will be easier to discuss in the saving docs repo). I'll reply to some points which I can do quickly: Syncing is the easier to explain of the two: TWC has plugins that let it > sync changes to TiddlyWeb, a venerable Python-based TiddlyWiki serverside > we developed at Osmosoft. I’ve not tested it recently, but those plugins > should also enable TWC to save back to a TW5 server. > Sounds interesting. This is namely two-way syncing or one-way saving of separate tiddlers? In other words, can it be used for simultaneous editing of different tiddlers by different users so that they see the changes made by others without reloading TW? Is syncing tiddler-wise? Meaning that it can be used for what I've just described but cannot be used for simulteneous editing one same tiddler by different users? By the way, why saving the whole HTML is not easier to explain: is it very different from what TWC uses for ordinary saving? TW5 supports saving in two completely different ways: saving the entire > HTML file versus syncing individual tiddlers to a server. > Same question here: TW5 supports granulated (tiddler-wise) saving or two-way syncing? (which would be very cool) In TW5, saving is modular: there are eleven built into the core: > So each of them implements both "full-TW-saving" and "syncing"? Ok, I'll stop here for now. The other things have more nuances to describe/learn/discuss, I'll just leave unanswered quotes below. Best regards, Yakov. > Next and the main point of this thread is: both TWC and TW5 have many > supporting solutions (native apps, servers, browser plugins etc) for saving > but I'm not aware of docs that describe how actually one supports saving > (and may be more functionality) of both TWC and TW5. For instance, > MainTiddlyServer <https://yakovl.github.io/MainTiddlyServer/> could be > extended a bit so that it fits the php server gap > <https://github.com/TiddlyWiki/php-server>, but I (as a developer of TWC > extensions, MTS and maintainer of the core) am not really aware of how to > support TW5 correctly; likewise, the aTW > <https://github.com/TiddlyWiki/aTW> saver for Android supports only TW5 > (as far as I know), but I'd like to participate and support TWC as well, > yet I still didn't even get an answer > <https://github.com/TiddlyWiki/aTW/issues/1#issuecomment-459076550> about > the app status (and I suspect Simon may be reluctant to support TWC without > nice docs describing that). Finally, it seems every time one tries to > support saving in their software, they invent some new implementation which > (this diversity) is difficult to follow and the ecosystem is quite > difficult to maintain. > > > [TW5 2 saving ways] > > In TW5, saving is modular: there are eleven built into the core: > > - andtidwiki - for saving within the AndTidWiki Android app > - beaker - for saving within Beaker Browser > - download - for the fallback HTML5 compatible download saver > - fsosaver - for the Internet Explorer ActiveX saver > - manualdownload - a version of the download saver for older browsers > that don’t support the necessary HTML5 features > - msdownload - a version of the download saver for older Microsoft > browsers > - put - for saving to WebDav servers > - tiddlyfox - for saving under TiddlyFox > - tiddlyie - for saving to the Internet Explorer TiddlyIE extension > - twedit - for saving within TWEdit, an old iPad/iPhone app for > TiddlyWiki > - upload - for saving to TiddlySpot, or a PHP server running the > upload script > > The TiddlyFox saver talks to TiddlyFox via a hidden DIV and custom events. > > TiddlyFox itself has different handling for TW5 and TWC. For TW5, it uses > the hidden DIV and custom events, and things are reasonably elegant. For > TWC, TiddlyFox has to inject overrides to the saving functions which then > set up the hidden DIV and handle the custom events. > > > My proposal consists of 2 parts: > > 1. create a repo in the TiddlyWiki <https://github.com/TiddlyWiki/> > org @github (I'd like to be a member to contribute) > 2. start building docs > > For the second part, I now have a draft based on what I've learned from > the TiddlyFox sources, see it below the main post. I invite you to comment > it, correct where I'm wrong, fill the gaps, especially about TW5. > > Good stuff. I think you’ve captured the behaviour of the TiddlyFox plugin. > I’ve added comments to your questions. > > This surely can give false positives which seems to be ok; this also > doesn't support "pure store" format. > > I do remember the “pure store” format being discussed, but don’t recall > the details. If it reduces the HTML file to the storeArea, then the > heuristics above should work. > > > - MainTiddlyServer injects JS bits into the TWC code before serving it > and strips those back when saves "the whole thing" (it can also send save > increments); some modifications (like support of incremental saving) are > made as hijacks which are done in loadPlugins > <https://github.com/YakovL/MainTiddlyServer/blob/master/index.php#L467> > so that the store is ready > - server-side implementations (that inject JS into TW so that user > doesn't have to install a plugin into TW) also need to decide where to > inject JS, see next section > > I’m not familiar with the details of MTS, but it sounds like these > modifications would be done on the HTML file, rather than in the DOM. > Somewhat different techniques need to be used in those situations. > > > - not sure whether TiddlyFox adds JS on load or eariler (probably the > latter) > > I think that the code gets injected after the page has loaded: > > https://github.com/TiddlyWiki/TiddlyFox/blob/master/index.js#L30 > > > - neither I am sure of the exact moment tiddly-node-saver uses > <https://github.com/fallwest/tiddly-node-saver/blob/master/server.js#L27> > although the approach looks interesting > - same for TiddlyDesktop overwriting > > <https://github.com/Jermolene/TiddlyDesktop/blob/ffd0c7be4d4220a67211e3b18ece208d11d305e4/source/js/utils/saving.js#L19> > > TiddlyDesktop just implements the TiddlyFox protocol of the hidden DIV and > the custom events. > > As *for TW5*, I'm not really sure about the exact stack, but it seems for > saving, a handler of the "tiddlywiki-save-file" custom event should be > added. The handler should expect the "event" (first argument) to have > .path and .content properties which should hold (local?) path to the file > to save and (UTF-8?) content string of the file. Not sure if some sort of > "file-save-success" event is supported to notify the core that the file was > actually saved (and TiddlyFox actually has a bug to always report that TW > was saved, even if it was on a USB storage which was removed before saving, > because the injected saver always returns true > <https://github.com/TiddlyWiki/TiddlyFox/blob/eb4c0d9822cf8beffe27a7cacb3b00c2d18e3771/data/inject.js#L26> > ). > > As discussed above, TW5 has pluggable save handlers so the two approaches > to implementing saving are to either re-use one of the existing core savers > (e.g. as does TiddlyDesktop) or to add a brand new saver module (e.g. as we > do for Beaker Browser). > > Best wishes > > Jeremy > > > *Further considerations* > > There's a number of topics that should be covered as well, mostly after > those above are, but some of them will affect architecture and hence should > be considered anyway (can affect the core, especially in case of TWC; I'm > ready to adapt it to better architecture): > > - sync vs async saving > - full saving vs sending changes, shortcuts for backuping (sending > full file to backup vs sending "copy file A into B" message) > - protocol layer (shouldn't necessarily be uniform, but > recommendations can facilitate development of new tools) > - streaming changes via websockets and sync editing > - security consideratinos > - server logic (authorization & roles, ..) > - federation (including content from other TWs, saving it back) > - advanced features (image uploading, integration with git etc) > > > > -- You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/tiddlywikidev. To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/10ff87ed-548a-4ada-8bb9-e7a36a82b908%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
