Nice newsreader! Fast and does what it needs to do, and written in D. I
like that. :)
I'm currently writing an NNTP web frontend (reading and posting) for my
university. However it's written in PHP so it's not really fitting for a
new D homepage. But I'm curious how you do web programming with D. Do
you use CGI? How do you do all the HTTP stuff (parsing form data, etc.)
and templating?
But back to the NNTP reader:
# HTML formating
The work you put into formating messages as HTML is impressive. The
autodetection of source code could really come in handy. I found
[Markdown][1] to work relatively well with common Mails so the syntax
might contain a few good ideas for e.g. quotes, links, lists, etc.
[1]: http://daringfireball.net/projects/markdown/syntax
# Topic list
Right now you display the newest few messages on the newsgroup. Building
a topic list gets quite a bit more complex. To get a proper topic list
with pagination etc. I query the overview information of all (!)
messages in a newsgroup with the "over" command (the digitalmars.com
server supports the older "xover" which is the same). This contains the
message ID and the references header which can be used to built a
message tree. All messages on the root level of the tree are topics and
it's easy to get the number of replies and the latest reply. It a bit
tricky sometimes but all other algorithms I came up with tend to lose
some messages (e.g. of the topic post is deleted) or were even slower.
The overview also contains the subject and from header and some other
useful stuff. I suppose the current newsreader does something similar
without caching and this might be the reason why it is so slow.
This message tree and the overview information however can be cached
very easily. The tree can also be extended on the fly, e.g. check for
new messages with the newnews command and add them to the tree. This
might require some locking but at least in PHP flock() was sufficient
for that.
# Cache invalidation
The problem with the message tree cache or cached messages in general is
the invalidation. Looks like the digitalmars news server does not delete
that much messages so this might not be much of a problem. How do you
handle this right now?
# D website
I took a look at your current version of the D website
(http://arsdnet.net/d-web-site/). I really like the layout. Looks good
to get started with D. Just two small things:
- The compile and run button is a bit of a security risk. I was able to
read the /etc/passwd file for example. Maybe it's possible to lock down
the compiled binaries with SELinux. Denial of service attacks (e.g.
endless loops) might still be a problem though. We built an "online D
compiler" for a presentation at our university but didn't published it
because of these concerns.
- If you only display mails in the announcements which do not have a
"References" header you will only get mails that started a new topic.
This will filter out replies.
If you want some help I could do some stuff. I'm a bit short on time
right now but since I'm building a NNTP reader in PHP anyway I might be
able to help out with your D NNTP reader. I can also help with HTML and
CSS stuff if you want. Support for older browsers and older IE versions
if there is much traffic with these browsers or some minor design stuff
(I'm not that much of a designer though). I might also start to look
into SELinux…
Happy programming
Stephan Soller
On 31.01.2011 04:08, Adam Ruppe wrote:
In the other newsgroup, I've been talking about a little
web news program I've been writing as a spinoff of the
potential new homepage idea.
It's to the point where it is usuable, but still kinda buggy:
http://arsdnet.net/d-web-site/nntp/thread-index?
newsgroup=digitalmars.D
Source code: http://arsdnet.net/d-web-site/nntp.d
NOTE: it does /not/ automatically check for new posts. I have
to manually trigger that right now (I don't want it annoying
the news server automatically while still in the testing phase.)
It will lazily load a message on demand though if you know
it's message ID:
http://arsdnet.net/d-web-site/nntp/get-message
Get it from the Message-ID header in the post.
Anyway, here's the features:
a) It isn't god awful slow. The PHP web news currently on digital
mars, as best as I can tell, actually polls the news server every
time you go to it's index! This does aggressive local caching.
b) It actually lets you select text...
OK, if I list every annoyance with the current web news, I'll
never stop. Moving on to new things:
c) It tries to convert news posts to HTML, so the paragraphs
wrap to the browser, links work, quotes are put into the proper
tags for indentation, and it tries to auto-detect D code and
put it in a<pre> block - which my javascript can make inline
editable and runnable. Example:
http://arsdnet.net/d-web-site/nntp/get-message?
newsgroup=digitalmars.D&messageId=%
3Cmailman.1085.1296409409.4748.digitalmars-d%40puremagic.com%3E
With script disabled, you'll see the code in a different colored
block. With script enabled, you'll see an Edit button there
too.
d) It tries to convert HTML emails back to plain text. (Ironically,
so it can turn it back to html...) This gives uniformity across
the various mime types. Similarly, if the type is
multipart/alternative, it will only show the text version.
e) It also makes an attempt to preserve deliberate whitespace,
for things like ASCII art or purposefully short lines. If it
can't make heads or tails of it, it bails out and shows the
original message in a<pre> block for human consumption.
f) Tries to be fast and lean.
g) Written in D!
h) Already read messages is tracked by your browser - if the link
is visited, it puts up a different color url.
Coming as I find time:
a) References to bugzilla entries should be automatically
converted to links.
b) Viewing threads by date or by threaded view.
c) Posting with the option of automatic quoting.
d) Syntax highlighting of D code in posts.
e) Maybe, maybe links to documentation of functions referenced,
if I can find a good way to get them automatically. Integration
with my dpldocs.info site is the way I'd do it.
e) Any more ideas? I'm reluctant to add too much, but if I like
an idea - or if you want to write the code :) - I'll be open'
to adding it.
Known bugs:
Lots of content types aren't handled right and it ignores
character encoding.
It doesn't always recognize code. This would be ok, but if it
sees one line as code but doesn't include one of them, it would
confuse the reader. Example:
http://arsdnet.net/d-web-site/nntp/get-message?
newsgroup=digitalmars.D&messageId=%3Cii4lbj%242bes%241%
40digitalmars.com%3E
(Look for "auto str =")
The reason for this is it detects code lines by looking for
semicolons and open braces. It will call something a generic
<pre> if there's a lot of whitespace in it - figuring it is
probaby ascii art (if it thinks the whitespace has human
significance, it tries to preserve it), but it still isn't
a perfect detection function.
I'm open to ideas. We want to detect code, but not flag
regular English text.
I'm also open to graphical styling ideas. I put up a dark
theme here because the white was hurting my eyes, but I change
on if I like light or dark almost at random. (Depends on the room's
lighting conditions I think). But I didn't do any more graphic
setup other than the max-width.
Multiple color schemes is an idea I like.
BTW, as a fun fact, this post is about 1/4th the size of the
entire nntp.d code file!
