Re: Thank you for public-inbox!
Jonathan Nieder wrote: > Eric Wong wrote: > > Jonathan Nieder wrote: > >> Jeff King wrote: > > >>> I guess I just wonder if I set up a mirror on another domain, would > >>> anybody actually _use_ it? I'd think most people would just go to > >>> public-inbox.org as the de facto URL. > >> > >> If it's faster than public-inbox.org and you don't mind the traffic I > >> would send, then I'll use it. :) > > > > Is performance a problem on public-inbox.org for you? > > It's pretty fast. I'm just very, very picky about latency. ;-) Best way for good latency is to have a local mirror, but I guess Googlers still aren't allowed to run AGPL software? > It's good to know you're interested in which corner cases are bad. > The next time I have a noticeably slow page load, I'll contact meta@. Alright. It could also be a general datacenter/networking problem so https://status.linode.com/ (my VPS provider) is worth checking. > [...] > > I've also been sorta considering downgrading to a $5/month VPS > > (from a $20/month VPS) to force myself to pay more attention to > > performance while saving myself a few bucks. But I wouldn't get > > to dogfood on SMP, anymore... > > Sounds reasonable to me. If performance gets bad, that's just a > reason for people to help out (either with patches or e.g. with > donated VMs for hosting). > Speaking of the latter: what are your current resource requirements? Not too much; but could always be better on the software side. > E.g. which of the dimensions in [1] do you not fit into? > [1] https://cloud.google.com/free/docs/always-free-usage-limits#compute_name Dunno, I'm not seeing RAM, there. Depending on traffic, it's around 200MB per-public-inbox-httpd worker (2 workers for 2 cores) when there's a traffic surge on from popular sites. Memory usage is the biggest disappointment and only happens when Varnish can't read fast enough. Everything in the PSGI code is is streamed if possible(*). My goal is to maintain <50MB per worker process, but it could be tough in Perl5. Anyways $20/month gets me 4GB RAM (so I have way more than I need). CPU usage isn't even noticeable (only bursts) and I do other stuff on that server all the time. HDDs wouldn't work well at all and I've noticed differences based on SSD quality with Xapian. Storage for Xapian+SQLite is 4-5x what's in git, so for this list, it's under 7G total (but more will be needed for Xapian reindexing/compact and git repacking). (*) Individual messages for returning giant mboxes and threads are all lazily fleshed out from skeleton data structures as the client socket becomes writable (and quickly discarded after writing). Technically it's all compatible with any PSGI server, but all the streaming stuff is tailored to run on public-inbox-httpd. But there's also git-http-backend memory use which comes in bursts (bitmaps enabled, of course)
Re: Thank you for public-inbox!
Eric Wong wrote: > Jonathan Nieder wrote: >> Jeff King wrote: >>> I guess I just wonder if I set up a mirror on another domain, would >>> anybody actually _use_ it? I'd think most people would just go to >>> public-inbox.org as the de facto URL. >> >> If it's faster than public-inbox.org and you don't mind the traffic I >> would send, then I'll use it. :) > > Is performance a problem on public-inbox.org for you? It's pretty fast. I'm just very, very picky about latency. ;-) It's good to know you're interested in which corner cases are bad. The next time I have a noticeably slow page load, I'll contact meta@. [...] > I've also been sorta considering downgrading to a $5/month VPS > (from a $20/month VPS) to force myself to pay more attention to > performance while saving myself a few bucks. But I wouldn't get > to dogfood on SMP, anymore... Sounds reasonable to me. If performance gets bad, that's just a reason for people to help out (either with patches or e.g. with donated VMs for hosting). Speaking of the latter: what are your current resource requirements? E.g. which of the dimensions in [1] do you not fit into? Thanks, Jonathan [1] https://cloud.google.com/free/docs/always-free-usage-limits#compute_name
Re: Thank you for public-inbox!
On Thu, Aug 30, 2018 at 07:20:49AM +, Eric Wong wrote: > > At the very least, I think if we plan to reference without an http URL > > that we would use something like URI-ish, like . That gives > > tools a better chance to say "OK, I know how to find message-ids" > > (though I still think that it's much less helpful out of the box > > compared to an http URL). > > That would be awesome if somelike like could be a > standard and adopted (likewise with ). > > I haven't checked, but are there existing/similar RFCs? > Surely somebody has tried to get > adopted by now, right? There's https://tools.ietf.org/html/rfc2392. They even call it "mid:". :) I don't know of any git URI scheme, though it's similar in spirit to magnet: links. Those are a bit verbose, though, because they're really a meta-format for a bunch of different content-addressable schemes. -Peff
Re: Thank you for public-inbox!
Jonathan Nieder wrote: > Jeff King wrote: > > > I guess I just wonder if I set up a mirror on another domain, would > > anybody actually _use_ it? I'd think most people would just go to > > public-inbox.org as the de facto URL. > > If it's faster than public-inbox.org and you don't mind the traffic I > would send, then I'll use it. :) Is performance a problem on public-inbox.org for you? It should be pretty fast, but maybe there's corner-cases and I haven't had time to pay close attention, lately... I've also been sorta considering downgrading to a $5/month VPS (from a $20/month VPS) to force myself to pay more attention to performance while saving myself a few bucks. But I wouldn't get to dogfood on SMP, anymore...
Re: Thank you for public-inbox!
Jeff King wrote: > On Wed, Aug 29, 2018 at 10:02:43AM +, Eric Wong wrote: > > Anyways I hope to teach public-inbox to auto-linkify Message-ID-looking > > strings "" into URLs for domain-portability, > > (but it's ambiguous with email addresses). But yeah, I don't > > like things being tied to domain names. > > That would be neat, but I think it actually makes references less useful > in a lot of cases. URLs are universally understood, which means: > > - people who don't know about public-inbox can just follow the link >(and in fact, that's how they learn how useful it is!) > > - even for people who do know about it, they are likely to read mails >in their MUA. And most MUAs have some mechanism for easily following >a URL, but won't know how to auto-linkify a message-id. Heh, one of the (unstated?) goals of public-inbox is to educate the users on how Message-IDs (and email in general) works. And to that end... > So I too dream of a world where I can say "give me more information on > this identifier" and my tools search a peer to peer distributed hash > table for it. But I don't think we live in that world yet. More than dreaming, our goal should be to BUILD such a world :> After all, it was my intense dislike of centralization which drew me to DVCS and git in the first place. > At the very least, I think if we plan to reference without an http URL > that we would use something like URI-ish, like . That gives > tools a better chance to say "OK, I know how to find message-ids" > (though I still think that it's much less helpful out of the box > compared to an http URL). That would be awesome if somelike like could be a standard and adopted (likewise with ). I haven't checked, but are there existing/similar RFCs? Surely somebody has tried to get adopted by now, right?
Re: Thank you for public-inbox!
Jeff King wrote: > I guess I just wonder if I set up a mirror on another domain, would > anybody actually _use_ it? I'd think most people would just go to > public-inbox.org as the de facto URL. If it's faster than public-inbox.org and you don't mind the traffic I would send, then I'll use it. :) [...] > That would be neat, but I think it actually makes references less useful > in a lot of cases. URLs are universally understood, which means: > > - people who don't know about public-inbox can just follow the link >(and in fact, that's how they learn how useful it is!) I agree: please don't stop using URLs. Having the message-id in the URL is very useful for being able to migrate to another server. In an ideal world, we'd be habitually using URLs that reference a redirector, in the same spirit as https://www.kernel.org/lore.html. That way, there is a very restricted syntax to use (e.g. just message-ids) that is stable and can be reconfigured to redirect to another service as appropriate. In principle it also allows tricks like redirecting based on geography or making the redirector go to the user's preferred archive interface based on a cookie. Thanks, Jonathan
Re: Thank you for public-inbox!
On Wed, Aug 29, 2018 at 10:02:43AM +, Eric Wong wrote: > Jeff King wrote: > > I've thought about mirroring it to a public server as well, just for > > redundancy. But without the same domain, I'm not sure it would be all > > that useful as a community resource. > > I wouldn't get too attached to the domain, "public-inbox.org" is > too long for my tastes anyways. "peff.net/git/$MESSAGE_ID" > would actually be more user-friendly :> > > A generic Message-ID redirection/finding service would be good, > (maybe some DHT thing, but... has that taken off for git blobs, yet?) Yes, and I agree that the URL portability is one of the things I really love about public-inbox (after all, I do have my own archive and now I can follow people's public-inbox links into my very-fast local copy). I guess I just wonder if I set up a mirror on another domain, would anybody actually _use_ it? I'd think most people would just go to public-inbox.org as the de facto URL. > Anyways I hope to teach public-inbox to auto-linkify Message-ID-looking > strings "" into URLs for domain-portability, > (but it's ambiguous with email addresses). But yeah, I don't > like things being tied to domain names. That would be neat, but I think it actually makes references less useful in a lot of cases. URLs are universally understood, which means: - people who don't know about public-inbox can just follow the link (and in fact, that's how they learn how useful it is!) - even for people who do know about it, they are likely to read mails in their MUA. And most MUAs have some mechanism for easily following a URL, but won't know how to auto-linkify a message-id. So I too dream of a world where I can say "give me more information on this identifier" and my tools search a peer to peer distributed hash table for it. But I don't think we live in that world yet. At the very least, I think if we plan to reference without an http URL that we would use something like URI-ish, like . That gives tools a better chance to say "OK, I know how to find message-ids" (though I still think that it's much less helpful out of the box compared to an http URL). -Peff
Re: Thank you for public-inbox!
On Wed, Aug 29 2018, Andrei Rybak wrote: > On 2018-08-29 12:02, Eric Wong wrote: >> Anyways I hope to teach public-inbox to auto-linkify Message-ID-looking >> strings "" into URLs for domain-portability, >> (but it's ambiguous with email addresses). But yeah, I don't >> like things being tied to domain names. > > This would be very useful for people who use MUAs without > Message-ID lookup capabilities, even without domain-portability. FWIW Many MUAs have some hidden way to do this, for example to find the E-Mail I'm replying to (your E-Mail) on GMail: https://mail.google.com/mail/u/0/#search/rfc822msgid%3A3eb1c5e8-3e89-0d2e-30b1-339f38c4c703%40gmail.com I.e. rfc822msgid: Confusingly Google Groups accepts the same syntax, but will barf if you include the <>'s, but that's from memory, maybe I misrecall or they've fixed it.
Re: Thank you for public-inbox!
On 2018-08-29 12:02, Eric Wong wrote: > Anyways I hope to teach public-inbox to auto-linkify Message-ID-looking > strings "" into URLs for domain-portability, > (but it's ambiguous with email addresses). But yeah, I don't > like things being tied to domain names. This would be very useful for people who use MUAs without Message-ID lookup capabilities, even without domain-portability.
Re: Thank you for public-inbox!
Jeff King wrote: > I've thought about mirroring it to a public server as well, just for > redundancy. But without the same domain, I'm not sure it would be all > that useful as a community resource. I wouldn't get too attached to the domain, "public-inbox.org" is too long for my tastes anyways. "peff.net/git/$MESSAGE_ID" would actually be more user-friendly :> A generic Message-ID redirection/finding service would be good, (maybe some DHT thing, but... has that taken off for git blobs, yet?) Anyways I hope to teach public-inbox to auto-linkify Message-ID-looking strings "" into URLs for domain-portability, (but it's ambiguous with email addresses). But yeah, I don't like things being tied to domain names. I've also been considering setting up a parallel instance on 80x24.org to use the more-scalable "v2" repository format developed for https://lore.kernel.org/lkml/ Speaking of lore, Konstantin confirmed he'll be getting more vger lists up: https://public-inbox.org/meta/20180827205811.GA1611@chatter
Re: Thank you for public-inbox!
On Mon, Aug 27, 2018 at 04:25:13PM +0200, Johannes Schindelin wrote: > I would like to take five minutes to thank you for public-inbox. It is > invaluable for me in the meantime. And I think I will never be able to > thank you enough for it. Let me echo that appreciation. I have always kept my own archive of the list, but it's so valuable to have stable URLs for communicating with other folks (and I find the general design of public-inbox _way_ more useful than gmane ever was). > P.S.: FWIW I added a mirror of public-inbox to > https://git-for-windows.visualstudio.com/git.public-inbox, so that my > automated tasks, as well as my playing around, does not stress your server > too much. I've thought about mirroring it to a public server as well, just for redundancy. But without the same domain, I'm not sure it would be all that useful as a community resource. Eric, let me know if there's something there that would help (e.g., putting more servers in DNS round-robin). -Peff
Re: Thank you for public-inbox!
You're very welcome, Johannes. And I'm hoping to have a few more goodies live this fall/winter for public-inbox :>
Thank you for public-inbox!
Hi Eric, I would like to take five minutes to thank you for public-inbox. It is invaluable for me in the meantime. And I think I will never be able to thank you enough for it. Just a couple of things where it is super useful to me: - Recently, my mail provider started dropping mails left and right. They might even be addressed to me, and they never arrive in my inbox (and not even in the spam folder: they just never arrive). I spent some ten hours this past weekend to script identifying the mails in public-inbox that I never received, and to weed through them. It seems I missed some 4,000 mails. Thanks to you, I now saw those mails (although I had to delete most of them after reading only their subject, in the interest of time). - Many a times when my automated builds identified a problem with the test suite, it was a lot quicker to use public inbox to identify the mail to respond to than to use my mail program. - Sometimes, I find myself in want of replying to a past patch, but back in the day when it was sent I thought it would not be interesting to me, so I deleted it. With public-inbox, I can easily get it in raw format, i.e. put it back into my inbox so I can reply. - I cannot tell you how many times I send a link to public-inbox to my colleagues rather than forwarding a mail, because the former will give them more context (and also semi-live updates in case somebody replied to said mail after I sent the link). A couple of things in the future that public-inbox will make possible for me: - You probably are aware of my GitGitGadget endeavor, a project similar in aim to SubmitGit, but a lot more integrated with the GitHub experience (and not requiring you to hand over your mail sending credentials to AWS). One particular feature I found myself really wanting in SubmitGit (but not possible due to its one-way design): I want my Pull Requests to be closed once the patches are integrated into git.git's `master` branch. While this feature is not available in GitGitGadget yet, I am well underway there. I already have a notes ref (`commit-to-mail`, available via https://github.git/gitgitgadget/git) that annotates commits in git.git with the Message-ID of the corresponding mail. By "I have", I mean: there is an automated task that uses public-inbox to keep that ref up to date. I also have an accompanying `mail-to-commit` notes ref that maps Message-IDs back to the corresponding commit in git.git. That notes ref "annotates" the (non-existing) blob you get when piping the Message-ID with a trailing newline to `git hash-object`. Again, this is information that would be absolutely unobtainable without public-inbox. - Related, I want to annotate the GitHub Pull Requests handled by GitGitGadvget with the corresponding name of the branch in https://github.com/gitster/git. This requires that `mail-to-commit` I mentioned in the previous bullet point, and therefore would not be possible without public-inbox. - A feature I plan on introducing into GitGitGadget is to attach comments to the GitHub Pull Request when anybody replies to the patch thread sent out by GitGitGadget. Also this feature would be impossible without public-inbox. - Another really useful feature I plan on introducing is to attach comments to those PRs whenever a What's Cooking is talking about the corresponding branch. Once again, would be impossible without public-inbox. So thank you, thank you, thank you, for public-inbox! Ciao, Dscho P.S.: FWIW I added a mirror of public-inbox to https://git-for-windows.visualstudio.com/git.public-inbox, so that my automated tasks, as well as my playing around, does not stress your server too much.