Re: [hackers] Rebuilding the foundations of NNTP
On Mon, Jul 28, 2003 at 03:11:23PM -0700, Kurt Cagle wrote: While I think that the commentary on NNTP is essentially correct -- we are rebuilding the foundation of NNTP, I want to raise a couple of issues that may justify just WHY such a rebuilding is necessary. I started working with NNTP back in 1992 ... it was in fact my first experience with the Internet, prior in fact to HTTP/HTML. At the time, NNTP was small, largely free, was run by a coterie of competent amateurs for the love of the medium and was very much devoted to handling the issues associated with maintaining threading across the boundaries of the nascent Internet. Please note that you're conflating NNTP (a transport mechanism) and Usenet (a messaging network which is transported by, *among other things*, NNTP). In a decade, the threats that have been lingering at the edge of e-mail has pretty much devoured NNTP. NNTP is difficult to moderate, difficult to search, difficult to archive, difficult to set up. If you want NNTP access, you often have to pay extra from your ISP, and there is no guarantee that the newsgroups that YOU need are going to be available via the server. The bulk of material circulating on Usenet is porn span, sent not by legitimate users of the servers but by companies that seem to feel that extreme (and typically disgusting) acts of sexual display will drive people to their sites. The high volume and poor archiving formats also insure that newsgroups are short memory archives at best. Finally, the role of the web has changed enough that most people are simply not aware that Usenet exists, even in those cases where it is available. You're assuming that I meant that you should become part of Usenet, and I wasn't. I was merely observing that, for the category of message transport that appeared to be in question, it seemed that NNTP was a better mechanism than RSS. *Some* of the things you want to move around *would* be better served by RSS, but not forum traffic -- which I understood to be the issue on point; was I wrong? Contrast that to what's going on with the current Drupal modules and RSS syndication. I've written chapters in a couple of books on RSS, and consequently have had a lot of chance to think about what exactly this medium is. RSS is significant in that it provides a way to aggregate links and associate that aggregation with some form of editorial filtering and annotation. Why is that important? In great part because it is a function which currently is not done very well within the confines of web pages. Many web pages contain links and editorial content on those links, but in most cases such information is not terribly filterable, is reliant upon webmasters remaining on top of their link pages on a regular basis (something that very seldom occurs in practice) and such feeds cannot be merged together to provide a large stream of aggregation. In other words, the meta-content that Web Pages are able to offer are far less than what RSS can do. Stipulated. But off my point. A Drupal node can be thought of as a distributor of RSS feeds of varying types, which may or may not also be a transport mechanism for content itself. In most cases RSS is most efficient when the only payload information it does carry is abstracts of contents and linkages, perhaps with enough overhead in terms of production dates and authors to allow verification systems to work effectively. Precisely -- and it sounded to me like people were talking about syndicating *the user comments themselves* -- for which is it manifestly not suitable. RSS abstracts and categories can be archived and persisted, can be formatted any number of different ways with relatively little work and because of its XML base works well in web services environments. You don't need a specialized server to use it, which isn't true of NNTP, and you aren't dependent upon having to go through a community process to create a new newsgroup, minimizing the alt.* phenomenon. You don't need a specialized server to use RSS? I'm not sure I believe that's an accurate charaterization at either end, but certainly not at the receiving end... And if you're running *your own servers*, then the latter problem isn't an issue either. And if you were running your own top-level (dean.*), then it wouldn't be an issue anyway if you could get people to carry it -- but I wasn't really suggesting that. In this day and age of NNTP client retrieval, it's not as necessary to do that sort of thing as it was in the days when propagation was an issue. Cheers, -- jra -- Jay R. Ashworth[EMAIL PROTECTED] Member of the Technical Staff Baylink RFC 2100 The Suncoast Freenet The Things I Think Tampa Bay, Floridahttp://baylink.pitas.com +1 727 647 1274 OS X: Because making Unix user-friendly was easier than debugging Windows
Re: [hackers] Rebuilding the foundations of NNTP
Doing the aggregation / syndication stuff as NNTP doesn't make much sense to me. We are creating a web app, it should use web protocols. RSS is perfect for this kind of things. It forces us to create the network to be far simpler and open than if we did it with NNTP - and that is a very good thing. -Zack On Tue, 29 Jul 2003, Jay R. Ashworth wrote: On Mon, Jul 28, 2003 at 03:11:23PM -0700, Kurt Cagle wrote: While I think that the commentary on NNTP is essentially correct -- we are rebuilding the foundation of NNTP, I want to raise a couple of issues that may justify just WHY such a rebuilding is necessary. I started working with NNTP back in 1992 ... it was in fact my first experience with the Internet, prior in fact to HTTP/HTML. At the time, NNTP was small, largely free, was run by a coterie of competent amateurs for the love of the medium and was very much devoted to handling the issues associated with maintaining threading across the boundaries of the nascent Internet. Please note that you're conflating NNTP (a transport mechanism) and Usenet (a messaging network which is transported by, *among other things*, NNTP). In a decade, the threats that have been lingering at the edge of e-mail has pretty much devoured NNTP. NNTP is difficult to moderate, difficult to search, difficult to archive, difficult to set up. If you want NNTP access, you often have to pay extra from your ISP, and there is no guarantee that the newsgroups that YOU need are going to be available via the server. The bulk of material circulating on Usenet is porn span, sent not by legitimate users of the servers but by companies that seem to feel that extreme (and typically disgusting) acts of sexual display will drive people to their sites. The high volume and poor archiving formats also insure that newsgroups are short memory archives at best. Finally, the role of the web has changed enough that most people are simply not aware that Usenet exists, even in those cases where it is available. You're assuming that I meant that you should become part of Usenet, and I wasn't. I was merely observing that, for the category of message transport that appeared to be in question, it seemed that NNTP was a better mechanism than RSS. *Some* of the things you want to move around *would* be better served by RSS, but not forum traffic -- which I understood to be the issue on point; was I wrong? Contrast that to what's going on with the current Drupal modules and RSS syndication. I've written chapters in a couple of books on RSS, and consequently have had a lot of chance to think about what exactly this medium is. RSS is significant in that it provides a way to aggregate links and associate that aggregation with some form of editorial filtering and annotation. Why is that important? In great part because it is a function which currently is not done very well within the confines of web pages. Many web pages contain links and editorial content on those links, but in most cases such information is not terribly filterable, is reliant upon webmasters remaining on top of their link pages on a regular basis (something that very seldom occurs in practice) and such feeds cannot be merged together to provide a large stream of aggregation. In other words, the meta-content that Web Pages are able to offer are far less than what RSS can do. Stipulated. But off my point. A Drupal node can be thought of as a distributor of RSS feeds of varying types, which may or may not also be a transport mechanism for content itself. In most cases RSS is most efficient when the only payload information it does carry is abstracts of contents and linkages, perhaps with enough overhead in terms of production dates and authors to allow verification systems to work effectively. Precisely -- and it sounded to me like people were talking about syndicating *the user comments themselves* -- for which is it manifestly not suitable. RSS abstracts and categories can be archived and persisted, can be formatted any number of different ways with relatively little work and because of its XML base works well in web services environments. You don't need a specialized server to use it, which isn't true of NNTP, and you aren't dependent upon having to go through a community process to create a new newsgroup, minimizing the alt.* phenomenon. You don't need a specialized server to use RSS? I'm not sure I believe that's an accurate charaterization at either end, but certainly not at the receiving end... And if you're running *your own servers*, then the latter problem isn't an issue either. And if you were running your own top-level (dean.*), then it wouldn't be an issue anyway if you could get people to carry it -- but I wasn't really suggesting that. In this day and age of NNTP client retrieval, it's not as necessary to do that sort of thing as it was in the days when
Re: [hackers] Rebuilding the foundations of NNTP
On Tue, 29 Jul 2003, Ka-Ping Yee wrote: On Tue, 29 Jul 2003, zachary rosen wrote: Doing the aggregation / syndication stuff as NNTP doesn't make much sense to me. We are creating a web app, it should use web protocols. RSS is perfect for this kind of things. It forces us to create the network to be far simpler and open than if we did it with NNTP - and that is a very good thing. I didn't mean to start off such a big thread. But lot of the debate seems to miss the point, to me -- i wasn't arguing (or even suggesting!) that we should drop what we're doing and run NNTP servers instead. I was trying to call attention in the *transport* (NNTP), not the *format* (RFC822). Of course metadata and structure are useful. Let's not throw that away, All i'm saying is that we can look to NNTP for inspiration when we design our transport, since it was a fairly reliable and successful way to spread content around. And let's face it, folks: we *are* designing a transport. HTTP polling is not the whole picture. It can't be. There are still open questions about caching and fetching articles selectively. I agree completly Ping - this is very good advice. Jay made an important point with respect to RSS: Precisely -- and it sounded to me like people were talking about syndicating *the user comments themselves* -- for which is it manifestly not suitable. For better or worse, NNTP succeeded at that. Of course we can decide we're not going to do that kind of syndication, and set the issue aside. But let's *know* that we are making that decision when we make it. Yes and yes. I am all for looking at NNTP for inspiration / experience, and all for knowing the scope of what we are building. I am just against using NNTP over RSS. -Zack -- ?!ng
Re: [hackers] Rebuilding the foundations of NNTP
On Tue, Jul 29, 2003 at 12:10:35PM -0500, zachary rosen wrote: Doing the aggregation / syndication stuff as NNTP doesn't make much sense to me. We are creating a web app, it should use web protocols. RSS is perfect for this kind of things. It forces us to create the network to be far simpler and open than if we did it with NNTP - and that is a very good thing. The issue was mailing lists vs. web boards; you will note that I *said* that weblogg-y stuff should be syndicated by RSS. Cheers, -- jra -- Jay R. Ashworth[EMAIL PROTECTED] Member of the Technical Staff Baylink RFC 2100 The Suncoast Freenet The Things I Think Tampa Bay, Floridahttp://baylink.pitas.com +1 727 647 1274 OS X: Because making Unix user-friendly was easier than debugging Windows -- Simon Slavin, on a.f.c
Re: [hackers] Rebuilding the foundations of NNTP
I am sorry for misunderstanding what you guys where pushing for. I read the majority of this thread in a pretty stupored state, and then the rest of it i read quickly before I rushed off to lunch. I was under the mistaken impression that there was support for using the actual NNTP protocol over RSS, and was kinda miffed about it :) Good thing I was wrong. -Zack On Tue, 29 Jul 2003, Ka-Ping Yee wrote: On Tue, 29 Jul 2003, zachary rosen wrote: Doing the aggregation / syndication stuff as NNTP doesn't make much sense to me. We are creating a web app, it should use web protocols. RSS is perfect for this kind of things. It forces us to create the network to be far simpler and open than if we did it with NNTP - and that is a very good thing. I didn't mean to start off such a big thread. But lot of the debate seems to miss the point, to me -- i wasn't arguing (or even suggesting!) that we should drop what we're doing and run NNTP servers instead. I was trying to call attention in the *transport* (NNTP), not the *format* (RFC822). Of course metadata and structure are useful. Let's not throw that away, All i'm saying is that we can look to NNTP for inspiration when we design our transport, since it was a fairly reliable and successful way to spread content around. And let's face it, folks: we *are* designing a transport. HTTP polling is not the whole picture. It can't be. There are still open questions about caching and fetching articles selectively. Jay made an important point with respect to RSS: Precisely -- and it sounded to me like people were talking about syndicating *the user comments themselves* -- for which is it manifestly not suitable. For better or worse, NNTP succeeded at that. Of course we can decide we're not going to do that kind of syndication, and set the issue aside. But let's *know* that we are making that decision when we make it. -- ?!ng
Re: [hackers] Rebuilding the foundations of NNTP
Doing the mailinglist as Usenet is a very interesting idea. The problem is, obviously, spam. But - being able to quickly browse / hop around all the different mailing lists would be a very useful thing. I don't think we could use NNTP to do it though unless we used it to just mirror the mails. Could we make some central mail indexing service that signed up to all the mailinglists and aggregated / displayed them all in a reasonable fashion? Something of this sort is actualy being used right now to archive this mailing lists (see the links off the hack4dean mailinglist page). Any other ideas? -Zack On Tue, 29 Jul 2003, Jay R. Ashworth wrote: On Tue, Jul 29, 2003 at 12:10:35PM -0500, zachary rosen wrote: Doing the aggregation / syndication stuff as NNTP doesn't make much sense to me. We are creating a web app, it should use web protocols. RSS is perfect for this kind of things. It forces us to create the network to be far simpler and open than if we did it with NNTP - and that is a very good thing. The issue was mailing lists vs. web boards; you will note that I *said* that weblogg-y stuff should be syndicated by RSS. Cheers, -- jra -- Jay R. Ashworth[EMAIL PROTECTED] Member of the Technical Staff Baylink RFC 2100 The Suncoast Freenet The Things I Think Tampa Bay, Floridahttp://baylink.pitas.com +1 727 647 1274 OS X: Because making Unix user-friendly was easier than debugging Windows -- Simon Slavin, on a.f.c
Re: [hackers] Rebuilding the foundations of NNTP
On Tue, Jul 29, 2003 at 12:54:00PM -0500, zachary rosen wrote: The issue was mailing lists vs. web boards; you will note that I *said* that weblogg-y stuff should be syndicated by RSS. Doing the mailinglist as Usenet is a very interesting idea. The problem is, obviously, spam. But - being able to quickly browse / hop around all the different mailing lists would be a very useful thing. Well, if you run your own NNTP servers, it's likely to be *easier* to kill spam, I should think. I don't think we could use NNTP to do it though unless we used it to just mirror the mails. Could we make some central mail indexing service that signed up to all the mailinglists and aggregated / displayed them all in a reasonable fashion? Something of this sort is actualy being used right now to archive this mailing lists (see the links off the hack4dean mailinglist page). Bidirectional gating between newsgroups and mailing lists is *well* understood by now -- another advantage of Getting the Glue Right. As, for that matter, is indexing netnews traffic. And, indeed, if you're running the servers yourself, your indexing facility could provide news:// links that would permit users to avoid the impedance mismatch between the powerful Usenet-based toolsets and the usually-less-powerful (and almost always differing) web-front-end toolsets. And, to clarify again, my goal is to stamp out web-bulletin-boards as a tool for doing what is, essentially, netnews. They're usually not especially powerful, and they're *different* almost everywhere. They're inefficient to use, and worse inefficient to learn. You can't cache them locally, which means you're dependent on external sources for searching tools, and they're just not as evolved (task-wise) as even the Windows newsreaders, let alone the Unix ones. This is another of those split-constituency problems: I'd bet that the Dean-online audience will be close to 25% power-users, and such people tend to be opinion-leaders against their flock. If you can make life easier for them without making it appreciably more difficult for yourself, it's always A Good Thing. Cheers, -- jra -- Jay R. Ashworth[EMAIL PROTECTED] Member of the Technical Staff Baylink RFC 2100 The Suncoast Freenet The Things I Think Tampa Bay, Floridahttp://baylink.pitas.com +1 727 647 1274 OS X: Because making Unix user-friendly was easier than debugging Windows -- Simon Slavin, on a.f.c
Re: [hackers] Rebuilding the foundations of NNTP
If the news servers can sync with the mailing lists and the servers could be managed and payed for by in kind donations (unnoficial campaigners) then it sounds wonderful to me. Run with it man - hash it out on the wiki, get some devs, and build the sucker. thats my advice ;) -ZAck On Tue, 29 Jul 2003, Jay R. Ashworth wrote: On Tue, Jul 29, 2003 at 12:54:00PM -0500, zachary rosen wrote: The issue was mailing lists vs. web boards; you will note that I *said* that weblogg-y stuff should be syndicated by RSS. Doing the mailinglist as Usenet is a very interesting idea. The problem is, obviously, spam. But - being able to quickly browse / hop around all the different mailing lists would be a very useful thing. Well, if you run your own NNTP servers, it's likely to be *easier* to kill spam, I should think. I don't think we could use NNTP to do it though unless we used it to just mirror the mails. Could we make some central mail indexing service that signed up to all the mailinglists and aggregated / displayed them all in a reasonable fashion? Something of this sort is actualy being used right now to archive this mailing lists (see the links off the hack4dean mailinglist page). Bidirectional gating between newsgroups and mailing lists is *well* understood by now -- another advantage of Getting the Glue Right. As, for that matter, is indexing netnews traffic. And, indeed, if you're running the servers yourself, your indexing facility could provide news:// links that would permit users to avoid the impedance mismatch between the powerful Usenet-based toolsets and the usually-less-powerful (and almost always differing) web-front-end toolsets. And, to clarify again, my goal is to stamp out web-bulletin-boards as a tool for doing what is, essentially, netnews. They're usually not especially powerful, and they're *different* almost everywhere. They're inefficient to use, and worse inefficient to learn. You can't cache them locally, which means you're dependent on external sources for searching tools, and they're just not as evolved (task-wise) as even the Windows newsreaders, let alone the Unix ones. This is another of those split-constituency problems: I'd bet that the Dean-online audience will be close to 25% power-users, and such people tend to be opinion-leaders against their flock. If you can make life easier for them without making it appreciably more difficult for yourself, it's always A Good Thing. Cheers, -- jra -- Jay R. Ashworth[EMAIL PROTECTED] Member of the Technical Staff Baylink RFC 2100 The Suncoast Freenet The Things I Think Tampa Bay, Floridahttp://baylink.pitas.com +1 727 647 1274 OS X: Because making Unix user-friendly was easier than debugging Windows -- Simon Slavin, on a.f.c
