Hello back,
we are moving away from the original goal of this thread. If possible, I would like to focus back on the current matter, which is to extend org's existing link format to arbitrary metadata. Still, as this proposal for richer links really starts to make sense as part of an actual workflow, I'll explain more what I propose --- maybe I should have started with that (if you're curious it's explained in depth there https://github.com/Judafa/linkin-org). The "linkin-org" workflow is fundamentally different than standard link systems, as in obsidian or in the database workflow that Jean Louis mentions, in the sense that the link's metadata are not updated in the background, but *only when the user opens a link*. It acts in reaction to the user's actions instead of passively in the background --- which aligns with org's and emacs's ways of doing, and make the whole system scale well. The key idea is that linked files have ids in their names. For instance: filename--20250425T155433.txt and a linked line contains a commented id, for instance (in a python syntax): # [id:20250425T160020] link gets you there Technically, this turns your file system into a distributed hash table. When you try to open a link, 3 simple routines are run in a row until one succeeds (from fastest to slowest): 1. check that the path inside the link is correct, if yes then we're done 2. if the path is incorrect, try to greedily reconstruct the path by local id searches (which is fast even when run with elisp, and always works if the file was just renamed but not moved) 3. if all above have failed, perform an extensive search of the file's id inside the computer (or the directories marked for search) (slow with elisp, but fast by delegating to fd for instance). linkin-org proposes a fast, safe way to create links: store your file (call linkin-org-store in dired) and paste the link wherever. Doing so added an id to your file and moved it to a location that the extensive will not miss. With this method, you have truly reliable links at the cost of following some simple rules (see the 4 golden rules in linkin-org's readme) and seeing ids in your file names. In my view, no centralized solution is suited (or at least should not be the default) as either it breaks easily (try moving a file in your dropbox app. Once your laptop synced, centralized solutions break) or it needs dedicated suite of softwares that will take care of the centralized update, which is not in emacs's philosophy (and not in mine as well). Julien "Jean Louis" <[email protected]> writes: >> I didnt think of the niceties that databases offer so far, but that >> looks useful indeed. > > A database does not have to be PostgreSQL. It can be simple and > shareable. It can be an Emacs hash table. It can be an XML file. It can > be an Org file. It can be a SQLite file in your git repository. It does > not matter. > > What matters is that there is a centralized place for the metadata. A > single source of truth. A place where you can store arbitrary > metadata—not just a limited set of fields that fit in a link syntax. > > Inserting limited metadata is not arbitrary metadata. If you can only > fit a few fields in the link, you are not supporting arbitrary metadata. > You are supporting a limited set of fields that someone decided were > important. > > Arbitrary metadata means any field, any value, any structure, any > relationship. That cannot fit in a link. That requires a data store. > That requires a database—whether it is PostgreSQL, SQLite, or an Org > file with properties. > >> Let me explain what kind of link-based workflow I have in mind before >> saying what system is best suited. >> My main focus is to make links reliable, in the sense that when you >> follow a link, it always just works; in linkin-org, this is ensured >> with a decentralized id system etc, but that's irrelevant for the >> current matter (check out https://github.com/Judafa/linkin-org to know >> more). >> From the user's point of view, it should feel as if you merged org-mode >> with dired as your links (with file type) are access points as reliable >> as a file listed in dired. > > That means that principle I have explained fit your own philosophy > perfectly. > > I am just not sure if you are saying one thing, but not following it or > not seeing. > > It is architectural solution that links never break to have unique IDs > and any changes of link are managed centralized. That way your Org links > do not break. Let us say if website link is not valid any more, you > could edit centralized place of links (call it database), and then all > the Org links could point to archived version of the link, user doesn't > change each particular link in many Org files. > > I do not know about you, but I had thousands of Org files, one for each > person. Imagine the problem to edit each single link in those files. It > is quite different when you use centralized links and just reference > them by UUID or ID. > >> For a link (with file type) to work, you just need two things: the link >> and the file. >> Makes sense, trivial, and that's meant to be this way. > > Trivial for small file, single user, who has time for thinking without > practicing. > > For any user beyond that, but even much below power users, that system > you are proposing is shallow. > > Files can change. Is link going to work always? > > If files are indexed in the database and you move them in the database, > then file system is forgotten, and links will always work that way (for > as long as hard disk and computer works). > > If files do not have metadata (indexing) then when file is renamed or > moved, the link is gone and destroyed. > > Solution to keep links working long term is to have indexed list of > links and to reference them by their ID. > >> By extension, I see any third-party, centralized intermediary as a >> weakness towards reliable links. > > I don't know what you mean with third parties. I have not suggested > having a third party. Having central place of links means that you have > to own and control your central place of links. That is not third party. > > If you do not own the hard disk and indexing system of third parties, > such as archive.org then you are more subject to lose reliable links. > >> Correct me if I'm wrong, but in case the database is >> lost/corrupted/non-updated, then the links may not work anymore. > > Sure, but same can be said for your files, if they are lost, corrupted, > you will not have reliable links, that is not context of the principle > explained. > >> And this would be the worst outcome: you're left with years of notes >> full of links that are now useless. > > Same thing if your hard disk get corrupted. > > Backing up the database is far more simpler than backing up the file > system. > > The issue of backing up your digital data is not really the subject, > isn't it? > > Arbitrary data in links is fully solved on my side. I told you I have > unlimited information on links, truly arbitrary information, and I am > using the principle explained. > >> I'm primarily targeting a fully decentralized system, which fits best >> with the philosophy of org mode imo. > > I am trying to understand how "decentralized system" fit into arbitrary > metadata, but okay, maybe you mean you wish to have each link not > centrally indexed, and so all information stored in the simple link. > > Arbitrary data means unlimited fields, values, structures, > relationships, so that cannot fit into link, or in the filename, or in > plain text file, that needs data store. > >> This comes with a whole bunch of desirable niceties: you can >> move/update your linked file anywhere (dropbox app on your phone, >> whatever) without notifying a database, the link still works. > > That relates to file synchronization, not a knowledge management > workflow. > > Dynamic Knowledge Repository is invented by Doug Engelbart, so you > should look into his work. He has invented it exactly for the same > purpose you are describing. Maybe it is lot to read? > > But let me see practically, if I have my database, and it is accessible, > then I can move my files anywhere I wish and want and links will still > work. > >> I would say it may even combine neatly: one can use the id of a file in >> last resort to make sure the database cannot lose track of the file. > > For those files over several bytes, I keep their index with hashes. So > re-indexing would find those files. That is another architectural > principle that you could lose so that your links stay always same, even > if files is moved on file system or renamed, as long as it is not > changed. > > And if you wish to change the file, then maybe such change should occur > over the layer of the index, so that file is first recognized, changed, > and re-indexed after the change. That way you could arbitrary change > files and have links still working as long as you use their IDs. > >> PS: Some more minor remarks: >> - >>> Link creation or capturing should be instant. 1-3 seconds. >> The link is created automatically, obviously. >> - with a database, no easy way to share a data afaict. Otoh if the >> metadata is inside the link, then that's as simple as paste the link, >> put the file in attachement, send the email. > > Sharing with a database is trivial. You export the object to a > self-contained format—JSON, XML, or even an Org property drawer—and send > it. The recipient imports it into their own database. The link remains > the same. The metadata remains intact. The relationships remain > preserved. > > Alternatively, you can embed the metadata in the link at export time. > The database stores the metadata. The link is the identifier. When you > export for sharing, you expand the metadata into the link. When you > import, you extract it back into the database. > > This gives you the best of both worlds: your internal system is > queryable, versioned, and scalable, and your shared links are > self-contained. > > Your system cannot do the reverse. Once the metadata is embedded in the > link, you cannot extract it into a database without parsing every link. > You are stuck with embedded metadata forever. > > Just as you would need to "attach the file" to share the link inside of > Org file, so I also need to press few buttons and I could export > whatever links and share with people.
