[dspam-dev] Development of DSPAM
Hello all The last weeks here almost every one was talking about forking DSPAM. Well... Sensory Networks replied and now every thing is quite again. But I want to take the opportunity to remind those one wanting to fork DSPAM that DSPAM is a community project and that we all work together to get DSPAM ahead. But if I look at http://dspam.nuclearelephant.com/features.shtml I see a list of feature requests and almost no one is posting patches to get those features or working on implementing them. Where are all of those people willing to fork DSPAM? Where are the developers? Where is the progress from the community? I took my time and have implemented #32 and #33. I am still testing it but it looks okay so far. I as well took several days to fix the mysql_drv. Fixing that beast was not so hard but testing it is another issue. I have so far processed around 250'000 mails without one single crash and without one singe dspam_clean run. And while processing all the 250'000 mails the DSPAM deamon plus all dspam client instances together never went over 15MB memory usage in total. And all of this with 16M max_allowed_packet in MySQL and 12MB on DSPAM MaxMessageSize and in client/server mode DSPAM with local domain sockets. With the old driver I was not able to do that. The old driver had in some areas 50% over allocation of really needed memory. And on my setup I had some time over 70MB memory usage inside DSPAM. And I had crashing DSPAM client and server: can not initialize agent context, permission denied for accessing socket, gone MySQL server, etc. This is now all gone. Fixed :) I am still running my test and will wait until it is finished with the corpus. After that I will run another test with just 4M max_allowed_packet. Okay. Why this long mail? Well... I would love to see other people on the list implement stuff. I am sure that others are developers or can develop. What is holding you back to do things? I miss the time when DSPAM was getting better and better with each release and where you did not had to wait almost 1 year to get a new DSPAM release. So please all you out there using DSPAM and able to code: fire up your dev environment and start coding on DSPAM. Add new features. Fix old known bugs (if you don't know them, then ask here and I am sure you will get responses). Enhance DSPAM. Make it faster. Make it use less memory. Add new storage engines. Etc... // SteveB -- Pt! Schon vom neuen GMX MultiMessenger gehört? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger
Re: [dspam-dev] Development of DSPAM
Hello all,
I'm no active developer, nowadays I'm not using DSPAM any more because of
instability and lazyness of our users to train, but recently somebody asked me
for sending my "Resend To:" micropatch, may be others could be interested (see
attachement).
Marek
Byl 3.prosinec 2007 (Pondělí), když v 16:47:28 (CET) , Steve napsal(a):
> Hello all
>
> The last weeks here almost every one was talking about forking DSPAM. Well...
> Sensory Networks replied and now every thing is quite again.
>
> But I want to take the opportunity to remind those one wanting to fork DSPAM
> that DSPAM is a community project and that we all work together to get DSPAM
> ahead.
>
> But if I look at http://dspam.nuclearelephant.com/features.shtml I see a list
> of feature requests and almost no one is posting patches to get those
> features or working on implementing them.
>
> Where are all of those people willing to fork DSPAM? Where are the
> developers? Where is the progress from the community?
>
> I took my time and have implemented #32 and #33. I am still testing
> it but it looks okay so far.
>
> I as well took several days to fix the mysql_drv. Fixing that beast was not
> so hard but testing it is another issue. I have so far processed around
> 250'000 mails without one single crash and without one singe dspam_clean run.
> And while processing all the 250'000 mails the DSPAM deamon plus all dspam
> client instances together never went over 15MB memory usage in total. And all
> of this with 16M max_allowed_packet in MySQL and 12MB on DSPAM MaxMessageSize
> and in client/server mode DSPAM with local domain sockets. With the old
> driver I was not able to do that. The old driver had in some areas 50% over
> allocation of really needed memory. And on my setup I had some time over 70MB
> memory usage inside DSPAM. And I had crashing DSPAM client and server: can
> not initialize agent context, permission denied for accessing socket, gone
> MySQL server, etc. This is now all gone. Fixed :)
>
> I am still running my test and will wait until it is finished with the
> corpus. After that I will run another test with just 4M max_allowed_packet.
>
> Okay. Why this long mail? Well... I would love to see other people on the
> list implement stuff. I am sure that others are developers or can develop.
> What is holding you back to do things? I miss the time when DSPAM was getting
> better and better with each release and where you did not had to wait almost
> 1 year to get a new DSPAM release.
>
> So please all you out there using DSPAM and able to code: fire up your dev
> environment and start coding on DSPAM. Add new features. Fix old known bugs
> (if you don't know them, then ask here and I am sure you will get responses).
> Enhance DSPAM. Make it faster. Make it use less memory. Add new storage
> engines. Etc...
>
> // SteveB
> --
> Pt! Schon vom neuen GMX MultiMessenger gehört?
> Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger
--
Marek Drápal - http://marek.drapal.org/
diff -Naur dspam-3.6.8/src/agent_shared.c dspam-3.6.8-patched/src/agent_shared.c
--- dspam-3.6.8/src/agent_shared.c 2006-05-13 14:17:30.0 +0200
+++ dspam-3.6.8-patched/src/agent_shared.c 2006-06-13 21:28:31.0 +0200
@@ -718,7 +718,7 @@
*/
buffer * read_stdin(AGENT_CTX *ATX) {
- int body = 0, line = 1;
+ int body = 0, line = 1, to = 0;
char buf[1024];
buffer *msg;
@@ -774,8 +774,16 @@
if (buf[0] == 0)
body = 1;
-if (!body && !strncasecmp(buf, "To: ", 4))
+if (!body && !strncasecmp(buf, "To: ", 4) && !to)
process_parseto(ATX, buf);
+ if (!body && !strncasecmp(buf, "Resent-To: ", 11))
+ {
+ process_parseto(ATX, buf);
+ /* If To: header comes later than Resent-To: we don't want to
+ * overwrite
+ */
+ to = 1;
+ }
}
if (buffer_cat (msg, buf))
diff -Naur dspam-3.6.8/src/daemon.c dspam-3.6.8-patched/src/daemon.c
--- dspam-3.6.8/src/daemon.c 2006-05-13 14:17:30.0 +0200
+++ dspam-3.6.8-patched/src/daemon.c 2006-06-13 21:28:20.0 +0200
@@ -859,7 +859,7 @@
buffer * read_sock(THREAD_CTX *TTX, AGENT_CTX *ATX) {
buffer *message;
- int body = 0, line = 1;
+ int body = 0, line = 1, to = 0;
char *buf;
message = buffer_create(NULL);
@@ -896,8 +896,14 @@
if (_ds_match_attribute(agent_config, "ParseToHeaders", "on")) {
if (buf[0] == 0)
body = 1;
-if (!body && !strncasecmp(buf, "To: ", 4))
+if (!body && !strncasecmp(buf, "To: ", 4) && !to)
process_parseto(ATX, buf);
+ /* Overwrite To: parsing if Resent-To: is set */
+ if (!body && !strncasecmp(buf, "Resent-To: ", 11))
+ {
+ process_parseto(ATX, buf);
+ to = 1;
+ }
}
if (buffer_cat (message, buf) || buffer_cat(message, "\n"))
diff -Naur dspam-3.6.8/src/dspam.c dspam-3.6.8-patched/src/dspam.c
--- dspam-3.6.8/src/dspam.c 2006-05-30 17:03:55.0 +0200
+++
[dspam-dev] Re: [dspam-users] Development of DSPAM
Firstly I'd like to thank SN for the replies last week, we now know your alive and listening. If no reply had been made a fork would be in progress, but you now get a perfect chance to show us how a SN+Community effort can exist. I would truly love SN to be more active on this lists and show that a community fork is not needed. You can see that community supports a fork, not because we don't want to work with SN, but your not helping us to help yourselves. A community fork will happen (project espam) by the end of the year if the following points cannot be moved forward: - Central Repository - with community access - Central Bug Tracker - with community access - Fold in exist SN patches, Gentoo patches and any other patches - Release a Beta with the patches and options in so we can start testing - Feature Suggestion - with community access, and place to discuss wacky ideas (I have some) - Wiki (maybe Frank will donate the start of this?) - Example setups, docs etc - with community access - Reliable DNS Servers (the dspam website appears offline most of time) - Invite Jonathan back into the party (if he can), the project misses him - Give us some ideas of what SN is doing with dspam - Better mailing list storage/search function - Establish contacts with distribution specific maintainers With comments and direction on the following: Move towards better distribution standardisation (you have at least Gentoo, CentOS and FreeBSD actively here) Structure/Roles for those willing to help out. Seems like the project can be sub divided (particularly split away webgui) and lots of Doc + testers Release/Test schedule Roadmap/Discussion of future ideas/features Discuss the illegality's of copyright for those that contribute to SN+Community project Assurance of SN commitment to open source and that the community will not be froze out Take ownership and show us the way. If SN doesn't have the time/resource to commit to the dspam project right now, then i'm not adverse to the Compiz/Beryl/Compiz idea. The community is waiting to help make dspam a better product, that's all we care about. While I'm in a writing mood, I'd also like to thank everyone on the dpsam lists for keeping the project alive this long. My particular thanks goes to Steve for all the work with the Gentoo patches and those with name suggestions for the forked project. 'espam' takes the votes, even though I personally smile at PigeonHole each time i say it :-) Maybe a pigeon can feature in the logo... I await a positive Sensory Networks reply, Paul Mark Rogers wrote: Steve wrote: The last weeks here almost every one was talking about forking DSPAM. Well... Sensory Networks replied and now every thing is quite again. I was thinking the same thing. I was optimistic that the message from SN indicated that there might be some involvement from them (and not on an "early next year" timescale) but it's gone quiet again. My opinion is that a fork would be worthwhile, if only to merge back into the official source if/when SN wish to do so (or not if they decide not). Okay. Why this long mail? Well... I would love to see other people on the list implement stuff. I am sure that others are developers or can develop. What is holding you back to do things? I miss the time when DSPAM was getting better and better with each release and where you did not had to wait almost 1 year to get a new DSPAM release. I think I found dspam just too late to have witnessed this, but it's obvious from the passion people have about dspam that it was once there. As a coder I might be able to offer something, but my days of C coding are some time ago. I would very much like to look at the web interface though (although I'm happier in PHP than Perl). Maybe I could do some documentation too. So please all you out there using DSPAM and able to code: fire up your dev environment and start coding on DSPAM. Add new features. Fix old known bugs (if you don't know them, then ask here and I am sure you will get responses). Enhance DSPAM. Make it faster. Make it use less memory. Add new storage engines. Etc... What I'm missing here is a clear idea of where to put the resulting code. Do we fork? Can we at least have a single repository of patches and other enhancements? Would SN be hostile to a fork? It seems to me that they might benefit from it (they can merge the code back after all, and they don't seem to have the resource to push dspam forward at the moment). If they are hostile to a fork then these mailing lists may disappear... If you look at how Compiz forked to create Beryl, which later merged back into Compiz once it was stable, then there might be a model we can follow here.
[dspam-dev] [PATCH 1/2] make the CGI scripts set the charset to ISO-8859-1
The CGIs don't set the charset; this causes Firefox to guess the character
set wrong sometimes, for example when I get spam with the subject in
Chinese or Hebrew; now that we're talking about it, sanitizing the
characters in the sender and subject would also be a good idea, methinks.
The author or authors of this submission hereby release any and all
copyright interest in this code, documentation, or other materials
included to the DSPAM project and its primary governors. We intend this
relinquishment of copyright interest in perpetuity of all present and
future rights to said submission under copyright law.
---
admin.cgi |2 +-
dspam.cgi |2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/admin.cgi b/admin.cgi
index 8b1147f..1a37038 100755
--- a/admin.cgi
+++ b/admin.cgi
@@ -586,7 +586,7 @@ sub output {
print "Expires: now\n";
print "Pragma: no-cache\n";
print "Cache-control: no-cache\n";
- print "Content-type: text/html\n\n";
+ print "Content-type: text/html; charset=ISO-8859-1\n\n";
my(%DATA) = @_;
$DATA{'WEB_ROOT'} = $CONFIG{'WEB_ROOT'};
diff --git a/dspam.cgi b/dspam.cgi
index 513a6a2..159fb27 100755
--- a/dspam.cgi
+++ b/dspam.cgi
@@ -1418,7 +1418,7 @@ sub output {
print "Expires: now\n";
print "Pragma: no-cache\n";
print "Cache-control: no-cache\n";
- print "Content-type: text/html\n\n";
+ print "Content-type: text/html; charset=ISO-8859-1\n\n";
my(%DATA) = @_;
$DATA{'WEB_ROOT'} = $CONFIG{'WEB_ROOT'};
[dspam-dev] [PATCH 2/2] filtering in the history: all | spam | innocent | whitelisted
Filtering in the history: all | spam | innocent | whitelisted
This is #34 in the feature request pool.
The author or authors of this submission hereby release any and all
copyright interest in this code, documentation, or other materials
included to the DSPAM project and its primary governors. We intend this
relinquishment of copyright interest in perpetuity of all present and
future rights to said submission under copyright law.
---
dspam.cgi | 59 ++-
templates/nav_history.html |4 +++
2 files changed, 55 insertions(+), 8 deletions(-)
diff --git a/dspam.cgi b/dspam.cgi
index 159fb27..b0161fc 100755
--- a/dspam.cgi
+++ b/dspam.cgi
@@ -214,6 +214,11 @@ sub DisplayHistory {
$CONFIG{'HISTORY_PER_PAGE'} = 50;
}
+ my($show) = $FORM{'show'};
+ if ($show eq "") {
+$show = "all";
+ }
+
if ($FORM{'command'} eq "retrainChecked") {
foreach my $i (0 .. $#{ $FORM{retrain_checked} }) {
my ($retrain, $signature) = split(/:/, $FORM{retrain_checked}[$i]);
@@ -225,7 +230,7 @@ sub DisplayHistory {
system("$CONFIG{'DSPAM'} --source=error --class=" . quotemeta($retrain) . " --signature=" . quotemeta($signature) . " --user " . quotemeta("$CURRENT_USER"));
}
}
- redirect("$MYURL&history_page=$history_page");
+ redirect("$MYURL&show=$show&history_page=$history_page");
} else {
if ($FORM{'retrain'} ne "") {
if ($FORM{'retrain'} eq "innocent") {
@@ -233,7 +238,7 @@ sub DisplayHistory {
} else {
system("$CONFIG{'DSPAM'} --source=error --class=" . quotemeta($FORM{'retrain'}) . " --signature=" . quotemeta($FORM{'signatureID'}) . " --user " . quotemeta("$CURRENT_USER"));
}
-redirect("$MYURL&history_page=$history_page");
+redirect("$MYURL&show=$show&history_page=$history_page");
}
}
@@ -249,6 +254,10 @@ sub DisplayHistory {
my($time, $class, $from, $signature, $subject, $info, $messageid)
= split(/\t/, $_);
next if ($signature eq "");
+
+# not good to check for messages to show here, we're skipping
+# the retraining data so retrained messages won't show
+
if ($class eq "M" || $class eq "F" || $class eq "E") {
if ($class eq "E") {
$rec{$signature}->{'info'} = $info;
@@ -263,6 +272,12 @@ sub DisplayHistory {
} elsif ($messageid == ''
|| $rec{$signature}->{'messageid'} != $messageid
|| $CONFIG{'HISTORY_DUPLICATES'} ne "no") {
+
+ # skip unwanted messages
+ next if ($class ne "S" && $show eq "spam");
+ next if ($class ne "I" && $show eq "innocent");
+ next if ($class ne "W" && $show eq "whitelisted");
+
$rec{$signature}->{'time'} = $time;
$rec{$signature}->{'class'} = $class;
$rec{$signature}->{'from'} = $from;
@@ -385,9 +400,9 @@ sub DisplayHistory {
}
if ($retrain eq "") {
- $retrain = qq!As ! . ucfirst($rclass) . "";
+ $retrain = qq!As ! . ucfirst($rclass) . "";
} else {
- $retrain .= qq! (Undo)!;
+ $retrain .= qq! (Undo)!;
}
my($path) = "$USER.frag/$signature.frag";
@@ -441,22 +456,50 @@ _END
$DATA{'HISTORY'} .= "[";
if (($history_pages > 1) && ($history_page > 1)) {
my $i = $history_page-1;
- $DATA{'HISTORY'} .= " < ";
+ $DATA{'HISTORY'} .= " < ";
}
for(my $i = 1; $i <= $history_pages; $i++) {
if ($i == $history_page) {
-$DATA{'HISTORY'} .= " $i ";
+$DATA{'HISTORY'} .= " $i ";
} else {
- $DATA{'HISTORY'} .= " $i ";
+$DATA{'HISTORY'} .= " $i ";
}
}
if (($history_pages > 1) && ($history_page < $history_pages)) {
my $i = $history_page+1;
- $DATA{'HISTORY'} .= " > ";
+ $DATA{'HISTORY'} .= " > ";
}
$DATA{'HISTORY'} .= "]";
}
+
+ $DATA{'SHOW'} = $show;
+ $DATA{'SHOW_SELECTOR'} .= "Show: ";
+ if ($show eq "all") {
+$DATA{'SHOW_SELECTOR'} .= "all";
+ } else {
+$DATA{'SHOW_SELECTOR'} .= "all";
+ }
+ $DATA{'SHOW_SELECTOR'} .= " | ";
+ if ($show eq "spam") {
+$DATA{'SHOW_SELECTOR'} .= "spam";
+ } else {
+$DATA{'SHOW_SELECTOR'} .= "spam";
+ }
+ $DATA{'SHOW_SELECTOR'} .= " | ";
+ if ($show eq "innocent") {
+$DATA{'SHOW_SELECTOR'} .= "innocent";
+ } else {
+$DATA{'SHOW_SELECTOR'} .= "innocent";
+ }
+ $DATA{'SHOW_SELECTOR'} .= " | ";
+ if ($show eq "whitelisted") {
+$DATA{'SHOW_SELECTOR'} .= "whitelisted";
+ } else {
+$DATA{'SHOW_SELECTOR'} .= "whitelisted";
+ }
+ $DATA{'SORT_SELECTOR'} .= "";
+
&output(%DATA);
}
diff --git a/templates/nav_history.html b/templates/nav_history.html
index 99ab051..6595ca3 100644
--- a/templates/nav_history.html
+++ b/templates/nav_history.html
@@ -38,6 +38,10 @@ deliver any false positives that are still in your quarantine.
+
+
+ $SHOW_SELECTOR$
+
Type
RE: [dspam-dev] Re: [dspam-users] Development of DSPAM
Hi all Firstly, thanks to everyone for their views over the past week or so. It's quite clear that the Dspam community is heavily invested in the project. A few points: * The CVS is still up and running at cvs.nuclearelephant.com, and details on login can be found at http://dspam.nuclearelephant.com/download.shtml. * Patches are being folded in, and a new Beta will be forthcoming in CVS * The Feature Suggestion/Request system exists at http://dspam.nuclearelephant.com/features.shtml, however a combination of the Wiki (see below) and the mailing lists may serve this better * Frank's great Wiki at http://dspamwiki.expass.de/, seems like it contains much of what you're looking for. * The DNS servers for Dspam are constantly under attack - this is not under SN control. I would like to revive some more of the mirrors to mitigate this issue but am waiting to hear back from the owners of those domains. * A central Bug Tracker is certainly possible, this is taking second priority to getting the CVS back into shape Cheers Mick Johnson Sensory Networks > - Central Repository - with community access > - Central Bug Tracker - with community access > - Fold in exist SN patches, Gentoo patches and any other patches > - Release a Beta with the patches and options in so we can start > testing > - Feature Suggestion - with community access, and place to discuss > wacky > ideas (I have some) > - Wiki (maybe Frank will donate the start of this?) - Example setups, > docs etc - with community access > - Reliable DNS Servers (the dspam website appears offline most of time) > - Invite Jonathan back into the party (if he can), the project misses > him > - Give us some ideas of what SN is doing with dspam > - Better mailing list storage/search function > - Establish contacts with distribution specific maintainers > > > With comments and direction on the following: > > Move towards better distribution standardisation (you have at least > Gentoo, CentOS and FreeBSD actively here) > Structure/Roles for those willing to help out. Seems like the project > can be sub divided (particularly split away webgui) and lots of Doc + > testers > Release/Test schedule > Roadmap/Discussion of future ideas/features > Discuss the illegality's of copyright for those that contribute to > SN+Community project > Assurance of SN commitment to open source and that the community will > not be froze out > Take ownership and show us the way. > > > If SN doesn't have the time/resource to commit to the dspam project > right now, then i'm not adverse to the Compiz/Beryl/Compiz idea. The > community is waiting to help make dspam a better product, that's all we > care about. > > > While I'm in a writing mood, I'd also like to thank everyone on the > dpsam lists for keeping the project alive this long. My particular > thanks goes to Steve for all the work with the Gentoo patches and those > with name suggestions for the forked project. 'espam' takes the votes, > even though I personally smile at PigeonHole each time i say it :-) > Maybe a pigeon can feature in the logo... > > > I await a positive Sensory Networks reply, > > Paul > > > > > > > > > Mark Rogers wrote: > > Steve wrote: > >> The last weeks here almost every one was talking about forking DSPAM. > >> Well... Sensory Networks replied and now every thing is quite again. > >> > > > > I was thinking the same thing. I was optimistic that the message from > > SN indicated that there might be some involvement from them (and not > > on an "early next year" timescale) but it's gone quiet again. > > > > My opinion is that a fork would be worthwhile, if only to merge back > > into the official source if/when SN wish to do so (or not if they > > decide not). > >> Okay. Why this long mail? Well... I would love to see other people > on > >> the list implement stuff. I am sure that others are developers or > can > >> develop. What is holding you back to do things? I miss the time when > >> DSPAM was getting better and better with each release and where you > >> did not had to wait almost 1 year to get a new DSPAM release. > >> > > > > I think I found dspam just too late to have witnessed this, but it's > > obvious from the passion people have about dspam that it was once > there. > > > > As a coder I might be able to offer something, but my days of C > coding > > are some time ago. I would very much like to look at the web > interface > > though (although I'm happier in PHP than Perl). Maybe I could do some > > documentation too. > > > >> So please all you out there using DSPAM and able to code: fire up > >> your dev environment and start coding on DSPAM. Add new features. > Fix > >> old known bugs (if you don't know them, then ask here and I am sure > >> you will get responses). Enhance DSPAM. Make it faster. Make it use > >> less memory. Add new storage engines. Etc... > >> > > > > What I'm missing here is a clear idea of where to put the resulting > > code. Do we fork? Can we at le
Re: [dspam-dev] Re: [dspam-users] Development of DSPAM
On Dec 3, 2007 4:11 PM, Mick Johnson <[EMAIL PROTECTED]> wrote: > * The CVS is still up and running at cvs.nuclearelephant.com, and details on > login can be found at http://dspam.nuclearelephant.com/download.shtml. Another nice to have would be to have ViewVC setup so the CVS repo can be web browsable. > * Patches are being folded in, and a new Beta will be forthcoming in CVS Awesome! > * The Feature Suggestion/Request system exists at > http://dspam.nuclearelephant.com/features.shtml, however a combination of > the Wiki (see below) and the mailing lists may serve this better The Wiki seems like the best place for this now. It seems like the features.shtml page should be updated, unless you really want people to continue sending money to Jonathan? That doesn't seem right since supposedly isn't active in the project any more. > * A central Bug Tracker is certainly possible, this is taking second > priority to getting the CVS back into shape Would also be nice to have. What about one of the nice integrated wiki/bug-tracker/source-control software like Trac (http://trac.edgewall.org/)? -Dave
