Re: [dev] running a shortlink provider
Be careful about running a link shortener. They are very prone to abuse. They don't cost money to run because of bloat but because they require a lot of moderation. Spam is incessant. NearlyFreeSpeech bans very little, but treats link shorteners as equivalent to running a mail server on their service, because they are always found and exploited by spammers. They write: https://faq.nearlyfreespeech.net/section/policy/proxy#proxy >URL shorteners are, unfortunately, a lot more fun to write than they are to >maintain. If you want to set up a URL shortener for your own use, that's fine. >If you let the general public submit URLs to it, expect us to shut it down the >first time it gets exploited. (And it will get exploited.) Properly-run URL >shorteners aren't successful because they have the shortest or cleverest URL, >they're successful because they have a team of people working 24x7 both >proactively and reactively to prevent and mitigate abuse. If you have such a >team, and you want to run a public URL shortener on our service, please >contact us for special arrangements. If you don't have such a team, you'll >have to find another host that's less concerned about the Internet's welfare. Worth bearing in mind. Cheers, Miles.
Re: [dev] running a shortlink provider
On Wed, May 31, 2023 at 12:44:14PM +0300, Santtu Lakkala wrote: > On 25.5.2023 6.47, Spenser Truex wrote: > > There are a lot of bloated projects that I've found. Any suggestions? > > It uses symbolic links in a directory as a data backend. This should not > have any race issues, as link creation should be an atomic operation. Note > that the default is to have 4 character tokens, that are randomly generated, > so expect collisions if you have a lot of links to shorten. Tinkered with this a bit more, this "backend" is quite easy for a server, and I was able to patch the OpenBSD httpd to server the symlinks as redirects in about 20 lines of code, out of which 3/4 is for making it configurable. -- BR, Santtu
Re: [dev] running a shortlink provider
Hi, On 25.5.2023 6.47, Spenser Truex wrote: My internet friends and I have been wanting a shortlinks provider. How can I do this in a suckless way? I'd like to be able to host the shortlinks provider and easily let my friends point their own domains at my VPS. I used to just upload a HTML file with a redirect to the desired location in my webserver root. This doesn't work for a shared service though. There are a lot of bloated projects that I've found. Any suggestions? Out of curiosity, I wrote a reasonably suckless PoC version[0] with C and CGI, tested it to work with OpenBSD httpd and slowcgi. It uses symbolic links in a directory as a data backend. This should not have any race issues, as link creation should be an atomic operation. Note that the default is to have 4 character tokens, that are randomly generated, so expect collisions if you have a lot of links to shorten. Every GET request is expected to be a redirection request, and every POST is a link addition. GETs are identified by PATH_INFO, POST data can have a token variable to use pre-specified text (if available), in addition to the mandatory uri, successful addition returns the token part as text/plain (i.e. not a full address). Error reporting is minimal, but errors probably are handled. I have no plans to implement this further, so consider this abandoned from the get go; I can add patches if someone uncovers some of the gaping holes there must be somewhere. If someone should want to test it, and the non-existent documentation is too sparse, please contact me directly, no reason to pollute the list. HTH [0]: https://inzg.it/shurl/files.html -- Cheers, Santtu
Re: [dev] running a shortlink provider
On 23/05/25 07:19PM, Spenser Truex wrote: > I don't support this database heavy database stuff. A key-value pair > dataset would be enough. It's basically a perl one-liner. See (just a quick example): https://stackoverflow.com/a/43050919/184064 This is still not an ideal solution, but it illustrates some of issues which can arise. Having a website in PHP around 2001 has taught me that writing to a file in a HTTP environment filled with crawlers, bots, people holding F5 and slow clients, is vastly different than writing to a file on a single-user system. > If you have a markup language for your database it's bloat. I mentioned a database system, not a markup language. > Why use a hammer when you can beat the nail with your hand? Indeed, one just doesn't do the latter. signature.asc Description: PGP signature
Re: [dev] running a shortlink provider
On 23/05/26 07:52PM, syg wrote: > Hi, > > Just my two cents, but why not just store the link destination, even > better, the HTML with the correct re-direction, in a file named > according to the hash. You can then just serve it as-is. No lock > problem. No database required. O(1) complexity. > > Of course this is only valid for a personal, non-public use since that > strongly limits the number of links you can have. If you edit your HTTP server instead you can send 300 requests and avoid having a lot of unnecessary bandwidth usage. > > Also, this is trivially extensible to a pastebin. > > - syg > -- CAEE B377 FC82 BAF9 102C D22F C5CE D003 1AA8 E281 Spenser Truexhttps://equwal.com signature.asc Description: PGP signature
Re: [dev] running a shortlink provider
Hi, Just my two cents, but why not just store the link destination, even better, the HTML with the correct re-direction, in a file named according to the hash. You can then just serve it as-is. No lock problem. No database required. O(1) complexity. Of course this is only valid for a personal, non-public use since that strongly limits the number of links you can have. Also, this is trivially extensible to a pastebin. - syg
Re: [dev] running a shortlink provider
Thanks for the details! I have tough about hash maps but actually not about simultaneous requests... this seems quite bad tho. (if you have more than a few hundreds users?) I know it is insecure, but I would store everything in a txt file where each line is a link. Why is that insecure? Because you can just fetch every shorted links by "bruteforce". this is "insecure" in the sense that it is not "private". If you don't want your id to be consecutive, a tsv file would do the trick with a hash then the link. BINGO I suppose that this doesn't help with complexity tho...
Re: [dev] running a shortlink provider
On 23/05/25 10:29AM, Anthony wrote: > On 5/25/23 07:29, Страхиња Радић wrote: > > Perhaps the most minimal solution for keeping data would be TSV files, but > > they > > are not suitable for storing data entered from the web because of > > concurrency, > > so a "real" database would be needed. > > > What do you mean by, "because of concurrency"? Performance: A hash table would be O(1) because you lookup the hash and get it. For a text file you could only hope for O(log n) with bisecting. Locks: No need to lock a hashtable. Needed very much to lock a file though. I don't support this database heavy database stuff. A key-value pair dataset would be enough. It's basically a perl one-liner. If you have a markup language for your database it's bloat. > > I know it is insecure, but I would store everything in a txt file where each > line is a link. Why is that insecure? > > Then you can reach your link with http://myperfectsite.org/ Neat. > > If you don't want your id to be consecutive, a tsv file would do the trick > with a hash then the link. BINGO > > Is there a problem with this approach ? > Why use a hammer when you can beat the nail with your hand? -- CAEE B377 FC82 BAF9 102C D22F C5CE D003 1AA8 E281 Spenser Truexhttps://equwal.com signature.asc Description: PGP signature
Re: [dev] running a shortlink provider
On 23/05/25 10:29AM, Anthony wrote: > What do you mean by, "because of concurrency"? At any given moment, several HTTP clients can request posting data at the same time. They can hang indefinitely in the middle of sending data. If the process involves writing to a file, this can lead to data corruption or data loss. Even with a file lock, other such requests can be delayed or even denied. That's why a more sophisticated approach, possibly with caching, transactions and other methods which are present in database systems, is needed. signature.asc Description: PGP signature
Re: [dev] running a shortlink provider
On 5/25/23 07:29, Страхиња Радић wrote: Perhaps the most minimal solution for keeping data would be TSV files, but they are not suitable for storing data entered from the web because of concurrency, so a "real" database would be needed. What do you mean by, "because of concurrency"? I know it is insecure, but I would store everything in a txt file where each line is a link. Then you can reach your link with http://myperfectsite.org/ If you don't want your id to be consecutive, a tsv file would do the trick with a hash then the link. Is there a problem with this approach ?
Re: [dev] running a shortlink provider
2023-05-25 7:29 GMT+02:00, Страхиња Радић : > Perhaps the most minimal solution for keeping data would be TSV files, but > they > are not suitable for storing data entered from the web because of > concurrency, > so a "real" database would be needed. > Probably it can be duct-taped writing new entries/edits on separate random files to be merged, however it's a poor man's write-ahead journal so an sqlite file or more would be more consistent.
Re: [dev] running a shortlink provider
Highlight: Link shorteners always go down due to non-funding and bloatware backends. Let's have a suckless one that is cheap to run! On 23/05/25 07:02AM, Marcel Plch wrote: > On Thu, May 25, 2023 at 12:47:20AM -0300, Spenser Truex wrote: > > Hello, > > > > My internet friends and I have been wanting a shortlinks provider. How > > can I do this in a suckless way? I'd like to be able to host the > > shortlinks provider and easily let my friends point their own domains at > > my VPS. > > > > I used to just upload a HTML file with a redirect to the desired > > location in my webserver root. This doesn't work for a shared service > > though. > > > > There are a lot of bloated projects that I've found. Any suggestions? > > > > > > -- > > CAEE B377 FC82 BAF9 102C D22F C5CE D003 1AA8 E281 > > Spenser Truexhttps://equwal.com > > 1) VPS > 2) Pick favorite Web stack > 3) Make a website that takes the specified URL, stores the URL and its >hash as a key-value pair (possibly in a database if that's suckless >enough for you, directory+file structure is probably okay as well.) > 4) Store the hash value in base64 > > > I personally would go for Python/Django, I'm not sure how suckless that's > considered (probably not at all) but I found Django to be about the > least bloated fully functional web framework. Web is in a sad state so > I'm sure Unix gods are gonna reward you even if you write a minimalistic > web in Django. > > Django also uses sqlite by default as a database backend, so it's not even > a glass cannon. It knows that unless you bother to set something up, you > probably don't need something special. > > > I'm sure this suggestion is going to be either extremely helpful or > upsetting. Let me know what you picked. ^_^ If we are going with "write your own backend" I'll do it in Common Lisp since I like it. It's even more lindy than C. I guess the suckless lisp is LISP 1.5. I suppose C could make a decent backend too, although I'm not really up to the task of doing it with C. That being said, it's a lot of work for something that has been redone a million times already. I did some more research and found hda.me's codebase. It uses nginx and postgresql. I'm not sure I want to use all that, but if I could make a backend that is compatible for the F-Droid app then that would be neat. The feature of either making 3-character tiny links that disappear, or longer links that last a long time is very smart. I found a bloatware that lets you just choose whatever link you want, but I don't think I'd want to let anyone use my domain to make vanity links like mydomain.com/mydomains-owner-is-a-cunt The hda.me owner said he stopped funding it because he needed multiple VPSes and cloudflare service to avoid getting his data-holding VPS shut down for malicious use of the link shortener. This is avoidable by just using a decent VPS provider and doing routine full disk backups just in case. My intended use case is for each person to use their own domain to make their own links, not really as a public link shortener. -- CAEE B377 FC82 BAF9 102C D22F C5CE D003 1AA8 E281 Spenser Truexhttps://equwal.com signature.asc Description: PGP signature
Re: [dev] running a shortlink provider
On 23/05/25 07:02AM, Marcel Plch wrote: > 2) Pick favorite Web stack [...] > I personally would go for Python/Django, I'm not sure how suckless that's > considered (probably not at all) [...] > Web is in a sad state so You answered your own question. Web is bloat, in particular HTML itself as something derived from SGML. Suckless way to have a URL shortener public service would probably be to create a CGI script in rc/shell or a program in C. Perhaps the most minimal solution for keeping data would be TSV files, but they are not suitable for storing data entered from the web because of concurrency, so a "real" database would be needed. signature.asc Description: PGP signature
Re: [dev] running a shortlink provider
On Thu, May 25, 2023 at 12:47:20AM -0300, Spenser Truex wrote: > Hello, > > My internet friends and I have been wanting a shortlinks provider. How > can I do this in a suckless way? I'd like to be able to host the > shortlinks provider and easily let my friends point their own domains at > my VPS. > > I used to just upload a HTML file with a redirect to the desired > location in my webserver root. This doesn't work for a shared service > though. > > There are a lot of bloated projects that I've found. Any suggestions? > > > -- > CAEE B377 FC82 BAF9 102C D22F C5CE D003 1AA8 E281 > Spenser Truexhttps://equwal.com 1) VPS 2) Pick favorite Web stack 3) Make a website that takes the specified URL, stores the URL and its hash as a key-value pair (possibly in a database if that's suckless enough for you, directory+file structure is probably okay as well.) 4) Store the hash value in base64 I personally would go for Python/Django, I'm not sure how suckless that's considered (probably not at all) but I found Django to be about the least bloated fully functional web framework. Web is in a sad state so I'm sure Unix gods are gonna reward you even if you write a minimalistic web in Django. Django also uses sqlite by default as a database backend, so it's not even a glass cannon. It knows that unless you bother to set something up, you probably don't need something special. I'm sure this suggestion is going to be either extremely helpful or upsetting. Let me know what you picked. ^_^ -- # Marcel Plch signature.asc Description: PGP signature
[dev] running a shortlink provider
Hello, My internet friends and I have been wanting a shortlinks provider. How can I do this in a suckless way? I'd like to be able to host the shortlinks provider and easily let my friends point their own domains at my VPS. I used to just upload a HTML file with a redirect to the desired location in my webserver root. This doesn't work for a shared service though. There are a lot of bloated projects that I've found. Any suggestions? -- CAEE B377 FC82 BAF9 102C D22F C5CE D003 1AA8 E281 Spenser Truexhttps://equwal.com signature.asc Description: PGP signature
