Hi Mario > Would it be possible to create a file, that doesn't contain the TW core. ... > My links file would be about 300k instead of 2.5MByte.
Yes, you can use a modified version of the main HTML template that omits all the JS to create an "inert" TiddlyWiki which will still be read perfectly happily by the scraper. Having said that, probably the best performance improvement we can make would be for the scraper to record the last modified date of each scan, and thus be able to pass an If-Modified-Since header to subsequent requests to avoid re-downloading data that hasn't changed. > Probably even less, if I would use a different template, to create the output > file. > OR > Would it be possible to create a community-links.json file that you can > fetch. ... It would be several kByte. We could certainly update the scraper to optionally accept JSON. But, it's quite nice that the originating sites are human readable because it gives users of links.tiddlywiki.com the opportunity to get to know the contributors better. Best wishes Jeremy > > -mario > On Sunday, March 21, 2021 at 10:37:15 PM UTC+1 Jeremy Ruston wrote: >>> So no misunderstanding: >>> >>> My wiki with "link tiddlers" can really be any wiki that contains other >>> non-link stuff because only the tiddlers tagged $:/tags/Link are fetched by >>> the script, right? >> >> That's correct, but another consideration is that the wiki will be fetched >> very frequently, so it's polite to make it as small as possible. >> >> I will add stats for the size of the wikis that are fetched so we could >> identify wikis that are carrying an unreasonable overhead, but for the >> moment we probably don't need to worry about it too much. >> >> The most important consideration is probably making your workflow as smooth >> as possible so that you've got every incentive to keep posting the links... >> >> Best wishes >> >> Jeremy >> >> >>> >>> <:-) >>> >>> On Sunday, March 21, 2021 at 9:39:04 PM UTC+1 Jeremy Ruston wrote: >>>> Just to add that the aggregator is currently set to run every 17 minutes, >>>> so once you're registered like Mohammad you can keep adding links and >>>> watch them appear on the site very shortly afterwards without any >>>> intervention from me. >>>> >>>> Best wishes >>>> >>>> Jeremy >>>> >>>> -- >>>> Jeremy Ruston >>>> [email protected] >>>> https://jermolene.com >>>> >>>>> On 21 Mar 2021, at 20:36, Saq Imtiaz <[email protected]> wrote: >>>>> >>>>> This is the content from the TiddlyWiki toolmap from about 6 months ago, >>>>> where each entry is in a separate tiddler: >>>> >>>>> >>>>> https://saqimtiaz.github.io/sq-tw/temp/streams-tiddlytoolmap%20-2.html#TiddlyWiki%20toolmap >>>>> >>>>> We might be able to automate extracting and putting the links in the >>>>> correct field, renaming and tagging the tiddlers. >>>>> Each entry should ideally still be manually reviewed for accuracy and >>>>> completeness though. >>>>>> On Sunday, March 21, 2021 at 6:00:23 PM UTC+1 Jeremy Ruston wrote: >>>>>>> One of the obstacles we face as a community is that we don’t have a >>>>>>> good place for us to curate and share bookmarks to useful resources. >>>>>>> Dave Gifford has shown what's possible with his ToolMap, an invaluable >>>>>>> community resource. But it’s a huge challenge for any one person to >>>>>>> keep an undertaking like that up to date, let alone complete. >>>>>>> >>>>>>> So, I’m pleased to announce TiddlyWikiLinks, a community links >>>>>>> aggregator that allows us to collectively curate bookmarks: >>>>>>> >>>>>>> https://links.tiddlywiki.com >>>>>>> >>>>>>> The way it works is that multiple contributing editors each maintain >>>>>>> their own TiddlyWiki containing their bookmarks. These wikis are >>>>>>> registered with the system on GitHub, and then at intervals a script >>>>>>> retrieves all the links and aggregates them together into a new wiki. >>>>>>> >>>>>>> The above links are to a static rendering to help Google find things, >>>>>>> but the information is also available in TiddlyWiki format: >>>>>>> >>>>>>> https://links.tiddlywiki.com/wiki.html >>>>>>> >>>>>>> It’s easy to set up to be a contributing editor. The main requirement >>>>>>> is that bookmarks be kept as tiddlers tagged "$:/tags/Link" with an >>>>>>> “url” field containing the URL. So, for example, I’m “jermolene”, and >>>>>>> all my links are to be found at: >>>>>>> >>>>>>> https://links.tiddlywiki.com/contributors/jermolene/ >>>>>>> >>>>>>> My bookmarking wiki is here: >>>>>>> >>>>>>> https://jermolene-links.tiddlyhost.com >>>>>>> >>>>>>> If enough people volunteer to be contributing editors, we can spread >>>>>>> the workload and make something much more valuable than any of us could >>>>>>> do individually. I’d be happy if we had a handful of contributing link >>>>>>> editors making high quality bookmarks, but I think this system could be >>>>>>> scaled to at least a few hundred. >>>>>>> >>>>>>> If you’re interested, please reply here with the URL of your >>>>>>> bookmarking wiki and your choice of username. >>>>>>> >>>>>>> As ever, there’s a lot to do on the development side to take this >>>>>>> forward: >>>>>>> >>>>>>> * Improving the styling (consider the current colour scheme and styling >>>>>>> to be a placeholder!) >>>>>>> * Adding JavaScript-based interactivity (particularly search) >>>>>>> * Adding support for multiple languages >>>>>>> * Adding an RSS/Atom feed >>>>>>> >>>>>>> The development has required setting up another GitHub organisation in >>>>>>> order to get another GitHub Pages site: >>>>>>> >>>>>>> https://github.com/TiddlyWikiLinks/TiddlyWikiLinks >>>>>>> >>>>>>> (For those with long memories, this is, of course, a variation of the >>>>>>> "TWederation" archiecture we have discussed as a community for a long >>>>>>> time). >>>>>>> >>>>>>> I’ve been able to put this together in a few days thanks to all that >>>>>>> I’ve learned on my recent client projects. My personal goal for 2021 >>>>>>> remains to get a Community Plugin Library up and running for >>>>>>> TiddlyWiki. The link aggregator is a simpler proposition that should >>>>>>> give us some valuable experience for that initiative. >>>>>>> >>>>>>> Questions and thoughts welcome, >>>>>>> >>>>>>> Best wishes >>>>>>> >>>>>>> Jeremy. >>>>>>> >>>>>>> >>>>>> >>>>> -- >>>> >>>>> You received this message because you are subscribed to the Google Groups >>>>> "TiddlyWiki" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>> >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/tiddlywiki/782fea9c-d64f-4e2b-9e30-9501c351580bn%40googlegroups.com. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "TiddlyWiki" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/tiddlywiki/bb9bba43-adf0-4f86-ab4e-869ad7601de7n%40googlegroups.com. > > -- > You received this message because you are subscribed to the Google Groups > "TiddlyWiki" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/tiddlywiki/e29bec8a-b82a-4398-ac9a-30f25d45c659n%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "TiddlyWiki" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/59C1CDD8-C900-4976-BD1E-4552749F689F%40gmail.com.
