Re: [tw5] Re: Taking Node Server to the next level

[bimlas]

Cozy link: https://docs.cozy.io/en/tutorials/konnector/scrape-data/
 

> Another option is Cozy, which, if I understand it, is a free online cloud. 
> This does not require the PouchDB saver, but a unique one, but may be used. 
> If I have more time, I'll dig into the saver world more.
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/93aa4048-f47d-4e62-b19e-cb446e068782%40googlegroups.com.

Re: [tw5] Re: Taking Node Server to the next level

[bimlas]

Arlen, Jed,

Thank you for supporting the community with your knowledge! Big respect!

Database backends look really useful. If I understand it, maybe even 
PouchDB can be solved online by Glitch: https://glitch.com/~pouchdb-server 
(click on show to fetch some data).

Another option is Cozy, which, if I understand it, is a free online cloud. 
This does not require the PouchDB saver, but a unique one, but may be used. 
If I have more time, I'll dig into the saver world more.

I don't know how Bob is solved, but here is a tutorial that can easily wrap 
both Tiddly and saver, so you can run it without installing Node.

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/197c1a68-0336-4750-aaa3-41372ab15342%40googlegroups.com.

Re: [tw5] Re: Taking Node Server to the next level

[Arlen Beiler]

And here's the link for it:

https://github.com/Arlen22/TW5-storage-plugin

On Mon, Dec 2, 2019 at 1:49 PM Arlen Beiler  wrote:

> Well, Jed's adapter and everyone's thoughts so far inspired me to do a
> little exploring and I managed to put this little proof of concept
> together. It turned out I did not need to touch the client at all and only
> modify four routes on the server. What's nice is that the index is still
> generated as usual so you can turn off the plugin and make changes to the
> initial HTML document if you want, then turn it back on and all your
> database tiddlers come back again. Almost like a template. The plugin is
> inside the data folder for now, but the npm install needs to be done in the
> tiddlywiki root folder (the folder containing tiddlywiki.js). The readme
> has rather simple instructions and the npm install command that needs to be
> run.
>
> On Mon, Dec 2, 2019 at 10:38 AM Arlen Beiler  wrote:
>
>> I think in this case I'm rather aiming to build something new on top of
>> TiddlyWiki5. I may revisit the rest of the TiddlyWeb protocol at a later
>> time and take some ideas from it. However, I think that to build something
>> new it would need to be built using TiddlyWiki5 from the ground up, whereas
>> tiddlyspot was built for TWC. That's just how I work best. For someone else
>> to take a different approach would not bother me at all.
>>
>> On Mon, Dec 2, 2019 at 5:30 AM mauloop  wrote:
>>
>>> Why not to reuse Tiddlyweb/Tiddlyspace work?
>>>
>>>- It has MySQL backend
>>>- It is multiuser and allows concurrent editing
>>>- It has public and private spaces
>>>- It uses the standard Node.js sync-adaptor
>>>- It works with TWC as well
>>>
>>> IMO it is a great piece of software. Not easy to set up. I did it on a
>>> local VM just to give it a try and Chris kindly helped me to solve some
>>> problems I fell into. I always feel bad when I see that good software falls
>>> into oblivion. Maybe it is not to be taken as is, but it could be a good
>>> starting point. Just to not reinvent the wheel.
>>>
>>> Another thought is... this is an old paradigm. PMario's suggestion (DAT
>>> protocol) goes one step beyond, moving towards the p2p paradigm. I use
>>> Syncthing to keep my devices synced. Both (DAT and Syncthing, which, I
>>> think, uses QUIC behind the scenes) provides versioning, encryption and,
>>> last but not least, ... privacy!
>>>
>>> TWedereation goes this direction too, isn't it?
>>>
>>> I have no much developing skills to contribute, but I would be happy to
>>> help testing if needed.
>>>
>>> )+(
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "TiddlyWiki" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to tiddlywiki+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/tiddlywiki/cb3cdfd0-e05a-417a-a329-1281e2064ecb%40googlegroups.com
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/CAJ1vdSQyBDE-qMhT4UrUnFiwsKQxDTd-VpudxcOGnV6rCoz5HQ%40mail.gmail.com.

Re: [tw5] Re: Taking Node Server to the next level

[Arlen Beiler]

Well, Jed's adapter and everyone's thoughts so far inspired me to do a
little exploring and I managed to put this little proof of concept
together. It turned out I did not need to touch the client at all and only
modify four routes on the server. What's nice is that the index is still
generated as usual so you can turn off the plugin and make changes to the
initial HTML document if you want, then turn it back on and all your
database tiddlers come back again. Almost like a template. The plugin is
inside the data folder for now, but the npm install needs to be done in the
tiddlywiki root folder (the folder containing tiddlywiki.js). The readme
has rather simple instructions and the npm install command that needs to be
run.

On Mon, Dec 2, 2019 at 10:38 AM Arlen Beiler  wrote:

> I think in this case I'm rather aiming to build something new on top of
> TiddlyWiki5. I may revisit the rest of the TiddlyWeb protocol at a later
> time and take some ideas from it. However, I think that to build something
> new it would need to be built using TiddlyWiki5 from the ground up, whereas
> tiddlyspot was built for TWC. That's just how I work best. For someone else
> to take a different approach would not bother me at all.
>
> On Mon, Dec 2, 2019 at 5:30 AM mauloop  wrote:
>
>> Why not to reuse Tiddlyweb/Tiddlyspace work?
>>
>>- It has MySQL backend
>>- It is multiuser and allows concurrent editing
>>- It has public and private spaces
>>- It uses the standard Node.js sync-adaptor
>>- It works with TWC as well
>>
>> IMO it is a great piece of software. Not easy to set up. I did it on a
>> local VM just to give it a try and Chris kindly helped me to solve some
>> problems I fell into. I always feel bad when I see that good software falls
>> into oblivion. Maybe it is not to be taken as is, but it could be a good
>> starting point. Just to not reinvent the wheel.
>>
>> Another thought is... this is an old paradigm. PMario's suggestion (DAT
>> protocol) goes one step beyond, moving towards the p2p paradigm. I use
>> Syncthing to keep my devices synced. Both (DAT and Syncthing, which, I
>> think, uses QUIC behind the scenes) provides versioning, encryption and,
>> last but not least, ... privacy!
>>
>> TWedereation goes this direction too, isn't it?
>>
>> I have no much developing skills to contribute, but I would be happy to
>> help testing if needed.
>>
>> )+(
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "TiddlyWiki" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to tiddlywiki+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/tiddlywiki/cb3cdfd0-e05a-417a-a329-1281e2064ecb%40googlegroups.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/CAJ1vdSSe4-y4wR%3D6o_ihBmYe8E_bUqAgqT0tpb7LwaSpv7K9Uw%40mail.gmail.com.

Re: [tw5] Re: Taking Node Server to the next level

[Arlen Beiler]

I think in this case I'm rather aiming to build something new on top of
TiddlyWiki5. I may revisit the rest of the TiddlyWeb protocol at a later
time and take some ideas from it. However, I think that to build something
new it would need to be built using TiddlyWiki5 from the ground up, whereas
tiddlyspot was built for TWC. That's just how I work best. For someone else
to take a different approach would not bother me at all.

On Mon, Dec 2, 2019 at 5:30 AM mauloop  wrote:

> Why not to reuse Tiddlyweb/Tiddlyspace work?
>
>- It has MySQL backend
>- It is multiuser and allows concurrent editing
>- It has public and private spaces
>- It uses the standard Node.js sync-adaptor
>- It works with TWC as well
>
> IMO it is a great piece of software. Not easy to set up. I did it on a
> local VM just to give it a try and Chris kindly helped me to solve some
> problems I fell into. I always feel bad when I see that good software falls
> into oblivion. Maybe it is not to be taken as is, but it could be a good
> starting point. Just to not reinvent the wheel.
>
> Another thought is... this is an old paradigm. PMario's suggestion (DAT
> protocol) goes one step beyond, moving towards the p2p paradigm. I use
> Syncthing to keep my devices synced. Both (DAT and Syncthing, which, I
> think, uses QUIC behind the scenes) provides versioning, encryption and,
> last but not least, ... privacy!
>
> TWedereation goes this direction too, isn't it?
>
> I have no much developing skills to contribute, but I would be happy to
> help testing if needed.
>
> )+(
>
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tiddlywiki+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/tiddlywiki/cb3cdfd0-e05a-417a-a329-1281e2064ecb%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/CAJ1vdSQNcg7eEZjiFuc%2BdoxbBOrC2Xu%3DbQGkAdJMZekgZhOtTA%40mail.gmail.com.

Re: [tw5] Re: Taking Node Server to the next level

[Arlen Beiler]

PouchDB is a very good first step because of the way it is setup. SQLite
would also be a great first step for targeting SQL databases.

On Sun, Dec 1, 2019 at 9:30 PM TonyM  wrote:

> Jed,
>
> Great work, what a Wiz. I will have a closer look. With the pouchDB have
> any more persistence that local storage I wonder?
>
> Such node solutions are great but we must find an easy way to host such
> things online when needed.
>
> Regards
> Tony
>
> On Monday, December 2, 2019 at 10:06:26 AM UTC+11, Jed Carty wrote:
>>
>> After finding out that I have very little idea how syncadaptors work I
>> started playing with making a new one and ended up with a working pouchdb
>> syncadaptor.
>>
>> It is very rough but it works, it will create a database and save
>> tiddlers to it, delete them when appropriate and then load the tiddlers
>> when you open it up again.
>>
>> At the moment the implementation is just for one wiki, but extending it
>> to have multiple wikis on the same node process would be very straight
>> forward.
>> It has potential for some things people have asked for like being able to
>> search all tiddlers in all available wikis.
>>
>> As a note, this is not noteself, while it could be configured to connect
>> to a remote database, either pouchdb or CouchDB, it doesn't have any of the
>> nice features of noteself other than saving to a database.
>>
>> Considering how straight froward this was making similar adaptors that
>> work with the various flavours of sql databases wouldn't be too hard aside
>> from figuring out the appropriate database schema. That is something I have
>> absolutely no interest in doing but this may work as a reasonable reference
>> design for anyone who wants to make a sql database adaptor.
>>
>> There are instructions on how to use the adaptor in the readme, it is not
>> at all polished and requires node.
>>
>> https://github.com/OokTech/TW5-PouchDBAdaptor
>>
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tiddlywiki+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/tiddlywiki/952167f3-cc7f-44a7-8093-629b2b3ef1a3%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/CAJ1vdSTCypzZmZy_P%3DXnhxWFSJRXFknZ3EYt_LXrhnHi8B%2BVWA%40mail.gmail.com.

Re: [tw5] Re: Taking Node Server to the next level

[TiddlyTweeter]

TonyM wrote:
>
> I do have a possibly faulty distant memory of a tiddlywiki writing to a 
> Google sheet.
>

I *think* Mat did that ... hang-on ... 
yes: https://groups.google.com/d/msg/tiddlywiki/QcTOJCVXkDw/kS5VBW0ACQAJ

It uses Google Forms & Spreadsheets.

Best
TT

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/9cf5e7ef-0ab4-43c2-a1ee-5324b8d022ce%40googlegroups.com.

Re: [tw5] Re: Taking Node Server to the next level

[TonyM]

Jeremy

I do have a possibly faulty distant memory of a tiddlywiki writing to a Google 
sheet. I found it curious but it had no use case for me at the time. I will see 
if I can find anything.

We are spoilt for choice in some ways yet I feel we need to remove a few speed 
humps.

Regards
Tony 

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/d2cd70f7-c691-49c6-9687-7e98f3a315b7%40googlegroups.com.

Re: [tw5] Re: Taking Node Server to the next level

[Jeremy Ruston]

There have been quite a few experimental syncadaptors over the years. Some of 
them come back from a simple GitHub search:

https://github.com/search?q=syncadaptor

I went through the first couple of pages and found:

* an old syncadaptor for CouchDB: https://github.com/cjxgm/tearly/
* a syncadaptor for GitHub: https://ustuehler.github.io/tw5-github/ 
* another old syncadaptor for GitHub: 
https://github.com/kpe/tw5-github-syncadaptor/
* a syncadaptor for SoLID linked data stores (see https://solid.mit.edu/): 
https://github.com/linonetwo/solid-tiddlywiki-syncadaptor

Another syncadaptor that I'd love to see is one that stores tiddlers in an 
online Google Sheet. Done right, one would be able to seamlessly switch between 
editing the same data within TiddlyWiki and via the Google Sheet user interface.

Best wishes

Jeremy

--
Jeremy Ruston
jer...@jermolene.com
https://jermolene.com

> On 2 Dec 2019, at 07:29, Jed Carty  wrote:
> 
> 
> It is stored locally on the hard drive like other node wikis, the difference 
> is that this is stored in some database files instead of individual tiddler 
> files. It is not localStorage or a browser based solution.
> 
> If you use the browser based version of pouchdb it is generally built on top 
> of indexeddb or websql which are just localStorage. It doesn't have any 
> magical fixes and isn't any more persistent than the localstorage plugin. In 
> browser it is just localStorage with a slightly different interface and data 
> model on top of it.
> 
> To put this online you would end up just duplicating noteself I think.
> -- 
> You received this message because you are subscribed to the Google Groups 
> "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to tiddlywiki+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/tiddlywiki/d406b178-989e-4c43-a9e4-a2efb476bdc2%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/171F0E9E-D679-45C2-BBD3-B73EE336DC33%40gmail.com.

Re: [tw5] Re: Taking Node Server to the next level

[Arlen Beiler]

Most Node solutions that I am aware of are either request-based (AWS Lambda
and Cloudfront Workers), or VM based. There's also Heroku, but it doesn't
allow a persistent file system. However, the data storage structure is a
more serious bottleneck for innovation, I believe, and once we have a
working system, it's easier to move to the next step. The biggest thing I
see is the unpredictable nature of the file system as a predictable
per-tiddler storage system. While it is possible to load tiddlers from the
file system, I feel like some of the inherent drawbacks in this system are
preventing us from taking it to the next level.

Once we store tiddlers in a database, we can write an alternate server that
just stores client tiddlers straight to the database without going through
a $tw instance on the server. We'll still have use for the server to
bootstrap this system, but once setup, it can be fairly self-contained. In
that sense, perhaps taking the node server to the next level means becoming
smarter about the common use cases and writing software that can handle
them more efficiently. Not much has been done to make use of the server
itself -- most of the Node-side innovation is in other commands. The server
was recently rewritten to make it more flexible for server-based solutions,
which is useful. But I'm seeing a lot of client-side solutions that don't
need the server at all, except to store tiddlers and files.

In one sense it feels like a bit of a departure from the major
possibilities that the TiddlyWiki server presents. And for me that's a
little sad because I read source code and the TiddlyWiki server has
enormous potential to work along-side the TiddlyWiki client in the browser
for various solutions. But the general use-cases do not need this type of
functionality, and indeed are encumbered by it because I there is no good
way to tell whether a particular data folder needs the server-side
components or not.

Right now I could write a server using the filesystem adapter as my spec
that would serve data folders using Apache and PHP, but there is no good
way to tell whether any particular data folder might require the
server-side instance, so it has to be explicitly stated in some way, and
for that we need to come up with a new data folder format.

The data folder could be marked by the specific presence or absence of
certain plugins or other tiddlywiki.info properties, in order to properly
trigger an alternate format, but I think the best way would probably be
simply to add a property to tiddlywiki.info like "clientOnly": true. This
would allow existing data folders to be converted easily if the owner knows
it is safe to do so. It's not hard to implement this system with the
current data folders either, it just requires a similar amount of effort,
which may be necessary anyway.

So I guess that breaks this up into several parts.

   - First we need to "invent" client-only data folders and implement them.
   - Second, we need to attach a more robust storage system that isn't
   encumbered by the various file systems. This would allow us to implement
   multi-user and version control quite a bit easier.
   - Third, we need to implement this in PHP as an identical
   implementation.

I know it's a lot, but that's my summary of the what I'm seeing with what
we've discussed so far.

Arlen

On Fri, Nov 29, 2019 at 5:44 PM TonyM  wrote:

> Cont...
>
> Sorry. Mobile gg can be a pain.
>
> I understand Jed has done a lot of on securing bob when internet facing.
>
> So what I am asking is if node where more common on hosting and safe to
> open to the internet could we not focus on a smaller subset of technologies
> with similar outcomes?
>
> Of course we value a diversity of solutions.
>
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tiddlywiki+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/tiddlywiki/e7fd7b93-d44d-4de2-bbe5-63492ce18501%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/CAJ1vdSS3Z4J5asgfhG82aHK6%2BwRxVwR-kpH%3DvBuea%3DnBD2uBdg%40mail.gmail.com.

Re: [tw5] Re: Taking Node Server to the next level

[Arlen Beiler]

I'm kind of thinking that if the user is setting up MySQL themselves, then
they have control of the server and might as well be setting up
TiddlyServer or Bob anyway, unless MySQL is up their alley, in which case
it's no big deal. On the flip side, using PouchDB or SQLite instead of the
file system would allow some more advanced solutions such as multi-user
editing and version control to be built in much easier.

I'm hoping eventually to find a way to host wikis in TiddlyServer without
using the Node server instance, since it does consume a certain amount of
memory per wiki, and this is one step in that direction. Once that is
accomplished we could transparently write a PHP version of TiddlyServer
that could be used on any shared hosting. Once that happens TiddlyWiki
could become a serious contender as a self-hosted multi-user wiki.

But since my paradigm is to operate within the constrains of current
TiddlyWiki code, I have to take it one step at a time.

On Fri, Nov 29, 2019 at 1:54 PM Jed Carty  wrote:

> Mark,
>
> The benefit of PouchDB is that it can be installed using NPM, or packaged
> into something like BobEXE without having to do a system-wide installation.
> For a local installation PouchDB is much easier to set up than
> MySQL/MariaDb. If you didn't go the packaged route like BobEXE it would at
> worst be cloning a repo and typing `npm install` to install the node
> dependencies. Then you don't have to worry about setting up users or
> anything on the databases, it is all self-contained.
>
> It is as easy to set up as sqlite, and if there are any MySQL/MariaDB
> servers that are that easy to set up I would very much like to know because
> every time I set up a database I forget how to do it and I have to look up
> how to set up new users and permissions all over again.
>
> On a remote server it would be at most one extra command when installing
> node and tiddlywiki, and that is only if we don't have an install script
> set up.
>
> If you want something larger on a remote server, CouchDB has the same
> interface as pouchdb and is as easy to set up as MySQL/MariaDB in my
> experience.
>
> I don't know if many hosting providers give out of the box CouchDB
> installations, so that may be a problem, but I think that anyone in a
> situation to need something that heavy duty would be experienced enough to
> not have trouble with it.
>
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tiddlywiki+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/tiddlywiki/1fb383cf-b1d0-4cd0-9768-a6f315076b53%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/tiddlywiki/CAJ1vdSQUnaOTbruvDuwADqBsC__epJMJyLGY-Ujqxh2pTakM8A%40mail.gmail.com.