[spamdyke-users] Spam rejection statistics ?
People, I have been using SD since 2009 but have only been keeping decent records of spam that makes it through to my server since 2015: 20151,003 20163,734 20177,999 20183,566 20192,921 20207,463 202110,209 20227,997 so far As you can see I have become lazy about keeping the config files up to date in the last few years and more spam has been getting through. I have some issues with spam that _shouldn't_ be getting through which I will ask about later but my question now is: Is there a way of logging the emails that are getting rejected? I expect that the stuff that is getting through to my qmail setup is an order of magnitude or so less than what is being rejected but it would be good to have some hard stats on it . . Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Blocking variations on a "From: " field
Bucky, On 2020-09-29 00:19, BC via spamdyke-users wrote: On 9/28/2020 7:51 AM, Philip Rhoades via spamdyke-users wrote: You need to block by header contents as it offers more wildcards: https://www.spamdyke.org/documentation/README.html#HEADERS From:* Hmm . . I thought I had tried that - oh well, I will give it a shot! I use this technique successfully but found that a space was required, thus: From: * Ah . . I think I would have used: From:*https://spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Blocking variations on a "From: " field
Marcin, On 2020-09-28 23:22, Marcin Orlowski via spamdyke-users wrote: Philip Rhoades via spamdyke-users wrote on 28.09.2020 06:34: People, I have tried a few different options but mails like these are still getting through: From: "Mark Milton" I want to block all email addresses that start with "mmilton01" - I presume it is possible but I haven't had any success so far . . You need to block by header contents as it offers more wildcards: https://www.spamdyke.org/documentation/README.html#HEADERS From:* Hmm . . I thought I had tried that - oh well, I will give it a shot! Thanks! Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] Blocking variations on a "From: " field
People, I have tried a few different options but mails like these are still getting through: From: "Mark Milton" I want to block all email addresses that start with "mmilton01" - I presume it is possible but I haven't had any success so far . . Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] Can I get SD going with IndiMail
Sam, I am gradually getting organised to change my netqmail installation over to IndiMail: http://www.indimail.org but have struck problems with getting SD working with it. It looks like SD is hard-coded to expect stuff to be in: /var/qmail What files does SD need from qmail? Is there a non-SMTP invocation which just takes mail on stdin and outputs the same on stdout and exists with a return value depending on whether the mail was spam or not spam? ie exits with some return value? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] ip-whitelist-entry Not Working
Eric, On 2018-06-04 04:41, Eric Broch via spamdyke-users wrote: can you have a comment (# philsdiscourse) on your IP whitelist entry line? maybe, remove '#philsdiscourse' and see what happens. Same problem - thanks anyway. P. On 6/3/2018 12:05 PM, Philip Rhoades via spamdyke-users wrote: People, I am trying to use my host qmail server as a relay for a docker container that is running on the host but mails are not being accepted - I have this in spamdyke.conf: ip-whitelist-entry=172.17.0.6 # philsdiscourse and I see this in the logs: Jun 4 03:53:59 prix spamdyke[28801]: FILTER_RDNS_MISSING ip: 172.17.0.6 Jun 4 03:53:59 prix spamdyke[28801]: FILTER_WHITELIST_IP ip: 172.17.0.6 entry: 172.17.0.6 # philsdiscourse but there is no ALLOW line that follows and the mail fails to be delivered - what am I missing? If I use swaks from the container, mail does get delivered OK but that is because spamdyke is being bypassed . . Thanks, Phil. -- Philip Rhoades "Life is too short . . we should be reducing suffering wherever we can while we explore the rest of The Universe - instead of destroying this beautiful and unique Pale Blue Dot". Douglas Adams (from "Hitchhiker's Guide to the Galaxy") “Space is big. Really big. You just won’t believe how vastly, hugely, mindbogglingly big it is. I mean, you may think it’s a long way down the road to the chemist’s, but that’s just peanuts to space.” PO Box 896 Cowra NSW 2794 Australia Web: http://philiprhoades.org E-mail: p...@philiprhoades.org Chat with my Avatar on FB here: https://www.facebook.com/PhiRhoChat -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] ip-whitelist-entry Not Working
People, I am trying to use my host qmail server as a relay for a docker container that is running on the host but mails are not being accepted - I have this in spamdyke.conf: ip-whitelist-entry=172.17.0.6 # philsdiscourse and I see this in the logs: Jun 4 03:53:59 prix spamdyke[28801]: FILTER_RDNS_MISSING ip: 172.17.0.6 Jun 4 03:53:59 prix spamdyke[28801]: FILTER_WHITELIST_IP ip: 172.17.0.6 entry: 172.17.0.6 # philsdiscourse but there is no ALLOW line that follows and the mail fails to be delivered - what am I missing? If I use swaks from the container, mail does get delivered OK but that is because spamdyke is being bypassed . . Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] unknown exit code from validation command, code 255: /usr/local/bin/spamdyke-qrv
People, I think I have seen this a couple of times recently: Apr 25 18:03:22 prix spamdyke[6851]: ERROR(filter_recipient_valid_inner()@filter.c:3085): unknown exit code from validation command, code 255: /usr/local/bin/spamdyke-qrv I haven't updated the code for a long time - maybe I should do that? I haven't tried to reproduce the problem yet . . Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org https://spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] SD Stats Report #3 - more spam getting through - CORRECTION
Sam, I forgot that even though I have "365" in my logrotate.conf file, I am still only getting 100 days of logs . . but since the last report I have kept about 11 months of spam messages that were not blocked by SD in a mail folder - I have adjusted the spreadsheet accordingly and now since the last report the successfully delivered spam has only increased by about 2x (from 0.4% to 0.8% of all the SpamDyke lines in the logs) - see below: On 2017-04-20 12:05, Sam Clippinger via spamdyke-users wrote: Nice spreadsheet! I don't have all the data you do, but just looking at my mail logs going back 1 month (excluding mailing list traffic), I gathered these reject/accept stats. I apologize if the formatting is messed up: Count Percent DENIED_RDNS_RESOLVE 72413 58.29 DENIED_RDNS_MISSING 26924 21.67 ALLOWED 6766 5.45 DENIED_SENDER_NO_MX 4730 3.81 DENIED_BLACKLIST_NAME 4630 3.73 DENIED_GRAYLISTED 3311 2.67 DENIED_RBL_MATCH 2059 1.66 DENIED_IP_IN_CC_RDNS 1936 1.56 TIMEOUT 776 0.62 DENIED_INVALID_RECIPIENT 457 0.37 DENIED_OTHER 127 0.10 DENIED_IP_IN_RDNS 71 0.06 DENIED_HEADER_BLACKLISTED 32 0.03 DENIED_SENDER_BLACKLISTED 6 0.00 DENIED_RECIPIENT_BLACKLISTED 1 0.00 Total 124239 For the recent report I get: 102417 FILTER_RDNS_MISSING 41317 ALLOWED 35222 DENIED_RDNS_MISSING 21230 DENIED_RBL_MATCH 19200 FILTER_RBL_MATCH 6164 FILTER_EARLYTALKER 1878 FILTER_INVALID_RECIPIENT 1878 DENIED_INVALID_RECIPIENT 1347 FILTER_RELAYING 1347 DENIED_RELAYING 1068 DENIED_SENDER_NO_MX 1053 FILTER_SENDER_NO_MX 764 FILTER_RDNS_RESOLVE 576 DENIED_RDNS_RESOLVE 472 TIMEOUT 290 FILTER_WHITELIST_IP 132 ERROR(output_writeln()@log.c:104): 28 FILTER_HEADER_BLACKLIST 28 DENIED_HEADER_BLACKLISTED 24 FILTER_SENDER_BLACKLIST 24 DENIED_SENDER_BLACKLISTED 6 FILTER_OTHER 6 DENIED_OTHER 2 ERROR(smtp_filter()@spamdyke.c:1721): 2 ERROR(nihdns_mx()@dns.c:1935): 1 ERROR(smtp_filter()@spamdyke.c:922): Clearly I don't run a high traffic server, but: - Numerically, the missing/unresolvable rDNS tests appear to be the most effective, though I haven't checked to see how many of those rejections were for valid email addresses. - For my own peace of mind, blocking subject lines with the header blacklist has been the only way to stop persistent spammers from reaching me via outlook.com [1] and gmail.com [2], which I'm not willing to block outright. Right. - The rDNS blacklist percentage appears to be very low but it's continually populated by my auto-blacklisting scripts and it's been very effective against organized groups (i.e. not botnets). Even though I rarely add to those scripts, I'm still amazed at how many new domains it catches every day. Are these auto scripts available? - I also use another set of scripts to automatically unsubscribe my users from "legitimate" mailing lists when they junk the messages (Gmail does this too). Since my users usually can't tell the difference between "real" spam and "legitimate" spam (and they don't care), those scripts cut down their junk mail without blocking constantcontact.com [3] and exacttarget.com [4] (and others like them). Right. To answer your questions, you can block "To: undisclosed-recipients" with the header blacklist filter, if that's really how it appears in the message headers. I'll give that a shot. Blocking emails with no "To" line in the header isn't something spamdyke can do right now, sorry! OK. Thanks! Phil. -- Sam Clippinger On Apr 18, 2017, at 9:36 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: People, It has been almost a year since the last report - here is the updated GD Spreadsheet: https://docs.google.com/spreadsheets/d/1GqinPR2mA0Jz-uTZ2zVJgutpiDl62HNbn2gWGNpd7Tk/pubhtml Unfortunately the amount of spam getting through the SD filtering, then seen by me and being moved to the spam folder has gone up almost five times since last year . . from the information I have now put more stuff in the black From and To lists . . I think the main problem is that my main email address is finding its way on to more and more spam lists . . How can I: - reject mails with no "To:" address - reject mails with a "To:" address of: "undisclosed-recipients" Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users Links: -- [1] http://outlook.com [2] http://gmail.com [3] http://constantcontact.com [4] http://exacttarget.com ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamd
[spamdyke-users] SD Stats Report #3 - more spam getting through
People, It has been almost a year since the last report - here is the updated GD Spreadsheet: https://docs.google.com/spreadsheets/d/1GqinPR2mA0Jz-uTZ2zVJgutpiDl62HNbn2gWGNpd7Tk/pubhtml Unfortunately the amount of spam getting through the SD filtering, then seen by me and being moved to the spam folder has gone up almost five times since last year . . from the information I have now put more stuff in the black From and To lists . . I think the main problem is that my main email address is finding its way on to more and more spam lists . . How can I: - reject mails with no "To:" address - reject mails with a "To:" address of: "undisclosed-recipients" Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Second SD Stats report - spamdyke-qrv SUCCESS!
People, On 2016-05-06 13:10, Philip Rhoades via spamdyke-users wrote: Sam, On 2016-05-06 02:50, Sam Clippinger via spamdyke-users wrote: You may need to recompile spamdyke-qrv with excessive output and run it with two "-v" flags to see the details you need. You don't need to actually install that recompiled copy; running it from the build folder should work just as well. Gives: . . QRV-EXCESSIVE(validate()@validate-qrv.c:818): INVALID RECIPIENT recipient: jackspr...@pricom.com.au resolved username: jackspratt Hmmm . . curious - that actually does what we expect. In the conf file I have: reject-recipient=invalid - am I still missing something? It seems spamdyke is not calling spamdyke-qrv? I finally worked out that the line: recipient-validation-command=/usr/local/bin/spamdyke-qrv had to be added to the conf file - shouldn't this have been in the conf file already but commented out? Thanks, Phil. -- Sam Clippinger On May 5, 2016, at 9:36 AM, Philip Rhoades <p...@pricom.com.au> wrote: Sam, On 2016-05-05 22:27, Sam Clippinger via spamdyke-users wrote: Very impressive numbers, thanks for sharing those! No worries - I plan to keep it up so I can see if gradually improving the spamdyking has an impact - my own previous setup had almost 100% blocking rate but with some false positives - it would be nice if I could get SD to that effectiveness but with no false positives! Out of curiosity, of the messages that were delivered, how did you judge if they were spam? Well the ones that make it through the system and are delivered and end up getting eyeballed and manually moved into the spam / phishing folder for counting / processing later. It sounds like the problem is that spamdyke-qrv is accepting messages to invalid addresses? Yes, and then when a delivery is tried the message gets bounced to the sender - which is normally bogus, so I end up getting a message: "Hi. This is the qmail-send program at pricom.com.au [1]. I tried to deliver a bounce message to this address, but the bounce bounced!" You can try running spamdyke-qrv manually with the "-v" flag (possibly twice) to see why it's deciding to allow the recipient. Something like this: spamdyke-qrv -v pricom.com.au [1] [1] jackspratt OK, that was one problem - I have never created a /var/qmail/users/assign file and built a /var/qmail/users/cdb file before . . but now, after going through that exercise, that command runs with no error or output and a delivery to jackspratt is still attempted . . Thanks, Phil. -- Sam Clippinger On May 4, 2016, at 4:39 AM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: People, Last year I reported some stats after I had been using SD for about a month and now I have a second set - unfortunately I forgot to increase the number of backlogs for logrotate and I lost a few months of data to compare delivered spam to but the latest stats are from 100 days of data: https://docs.google.com/spreadsheets/d/1GqinPR2mA0Jz-uTZ2zVJgutpiDl62HNbn2gWGNpd7Tk/pubhtml There were some changes to the conf file between sets of data but I didn't keep notes about changes and dates etc however it seems that the proportion of ALLOWED lines went down a little which suggests more spam was stopped - but conversely, the proportion of delivered spams compared to SD lines went up a little - which I don't quite understand . . Now I want to try and stop the delivered spams that have invalid email addresses - I have compiled and installed spamdyke-qrv OK and set "reject-recipient" to "invalid" but these spams are still getting through and then being bounced and since the return address is bogus I get a postmaster message that the bounce has failed eg for the address: jackspr...@pricom.com.au - suggestions? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users Links: -- [1] http://pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au Links: -- [1] http://pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Second SD Stats report
Sam, On 2016-05-05 22:27, Sam Clippinger via spamdyke-users wrote: Very impressive numbers, thanks for sharing those! No worries - I plan to keep it up so I can see if gradually improving the spamdyking has an impact - my own previous setup had almost 100% blocking rate but with some false positives - it would be nice if I could get SD to that effectiveness but with no false positives! Out of curiosity, of the messages that were delivered, how did you judge if they were spam? Well the ones that make it through the system and are delivered and end up getting eyeballed and manually moved into the spam / phishing folder for counting / processing later. It sounds like the problem is that spamdyke-qrv is accepting messages to invalid addresses? Yes, and then when a delivery is tried the message gets bounced to the sender - which is normally bogus, so I end up getting a message: "Hi. This is the qmail-send program at pricom.com.au. I tried to deliver a bounce message to this address, but the bounce bounced!" You can try running spamdyke-qrv manually with the "-v" flag (possibly twice) to see why it's deciding to allow the recipient. Something like this: spamdyke-qrv -v pricom.com.au [1] jackspratt OK, that was one problem - I have never created a /var/qmail/users/assign file and built a /var/qmail/users/cdb file before . . but now, after going through that exercise, that command runs with no error or output and a delivery to jackspratt is still attempted . . Thanks, Phil. -- Sam Clippinger On May 4, 2016, at 4:39 AM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: People, Last year I reported some stats after I had been using SD for about a month and now I have a second set - unfortunately I forgot to increase the number of backlogs for logrotate and I lost a few months of data to compare delivered spam to but the latest stats are from 100 days of data: https://docs.google.com/spreadsheets/d/1GqinPR2mA0Jz-uTZ2zVJgutpiDl62HNbn2gWGNpd7Tk/pubhtml There were some changes to the conf file between sets of data but I didn't keep notes about changes and dates etc however it seems that the proportion of ALLOWED lines went down a little which suggests more spam was stopped - but conversely, the proportion of delivered spams compared to SD lines went up a little - which I don't quite understand . . Now I want to try and stop the delivered spams that have invalid email addresses - I have compiled and installed spamdyke-qrv OK and set "reject-recipient" to "invalid" but these spams are still getting through and then being bounced and since the return address is bogus I get a postmaster message that the bounce has failed eg for the address: jackspr...@pricom.com.au - suggestions? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users Links: -- [1] http://pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] Second SD Stats report
People, Last year I reported some stats after I had been using SD for about a month and now I have a second set - unfortunately I forgot to increase the number of backlogs for logrotate and I lost a few months of data to compare delivered spam to but the latest stats are from 100 days of data: https://docs.google.com/spreadsheets/d/1GqinPR2mA0Jz-uTZ2zVJgutpiDl62HNbn2gWGNpd7Tk/pubhtml There were some changes to the conf file between sets of data but I didn't keep notes about changes and dates etc however it seems that the proportion of ALLOWED lines went down a little which suggests more spam was stopped - but conversely, the proportion of delivered spams compared to SD lines went up a little - which I don't quite understand . . Now I want to try and stop the delivered spams that have invalid email addresses - I have compiled and installed spamdyke-qrv OK and set "reject-recipient" to "invalid" but these spams are still getting through and then being bounced and since the return address is bogus I get a postmaster message that the bounce has failed eg for the address: jackspr...@pricom.com.au - suggestions? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] recipient-blacklist-file=FILE with RegExes?
Sam, On 2015-12-31 06:34, Sam Clippinger via spamdyke-users wrote: Ah... you're confusing the "sender" address with the "From" address. Dammit! . . I get caught with that every time I come back to look at this stuff . . The sender address is what appears in the logs. Of course . . The From address is what appears in the message headers and is also what you see in your mail client. The two are completely separate and spammers usually supply different (bogus) values for them. Right . . To block both of the examples you gave, add these lines to your sender-blacklist-file (not your header-blacklist-file): @brewster.com [1] @nice.com [2] Yes . . but I solved the "From:" and "Reply-to:" problem with a single file and globbing but I can't do that with the sender-blacklist-file as well . . I might set up one master file and do a nightly cron job that produces both of the needed files from the master file . . Thanks again! Phil. That should do it! More info here: http://www.spamdyke.org/documentation/README.html#REJECTING_SENDERS -- Sam Clippinger On Dec 29, 2015, at 11:54 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: People, I thought of starting a new thread but the question relates to this discussion so I thought I would revive it - see inline comments: On 2015-06-21 04:57, Philip Rhoades via spamdyke-users wrote: Sam, On 2015-06-21 03:12, Sam Clippinger via spamdyke-users wrote: Regex support is on the (rather lengthy) to-do list, but frankly it's not a very high priority -- there's a lot of low-hanging fruit that would be of much more benefit right now. Plus, since I'm not one of the 10 people in the world who completely understands regexes, I doubt I would actually use them myself; I'd rather add globbing support, which I do understand. :) OK, no worries - SD is going well so far so I may not need some of the mechanisms that I used in my own setup - we'll see how things go. spamdyke's header filter runs at connection time, as all of its filters do. If a header line matches a blacklisted pattern, the entire message is rejected (the sending server receives an error code, qmail never sees the message). Right - thanks for the clarification. One annoying spammer continues to get their mail through but I don't understand why - my header-blacklist-file includes these two lines in it: [FR][re][op][ml]*:*brewster.com* [FR][re][op][ml]*:*nice.com* but the first one works and the second one doesn't!: /var/log/maillog-20151230:Dec 29 17:08:43 prix spamdyke[15684]: DENIED_HEADER_BLACKLISTED from: smartdel...@brewster.com to: p...@pricom.com.au origin_ip: 23.253.183.234 origin_rdns: mail-183-234.mailgun.info auth: (unknown) encryption: (none) reason: /usr/local/bin/srejector2/spamdyke_blacklist_header.txt:11 /var/log/maillog-20151230:Dec 29 17:08:00 prix spamdyke[15609]: ALLOWED from: support.a...@nice.com to: mailer-dae...@pricom.com.au origin_ip: 192.114.148.4 origin_rdns: mailil.nice.com auth: (unknown) encryption: (none) reason: 250_ok_1451369280_qp_15628 I have even saved the file in vim a couple of times and restarted qmail a couple of times but no change in the behaviour - what could the explanation be? Thanks, Phil. On Jun 19, 2015, at 9:09 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: Sam, See inline comments: On 2015-06-20 11:53, Sam Clippinger via spamdyke-users wrote: You're correct spamdyke does not support regexes for any of its options, but you can use a wildcard in a sender or recipient white/blacklist file to match entire domains by prefixing the line with an @ symbol. For example: @example.com [1] [1] Yep, saw that - is it possible to support regexes in the future? Full documentation here: http://www.spamdyke.org/documentation/README.html#REJECTING_RECIPIENTS [2] [2] BUT! Be careful -- the "To" and "From" lines in the message header are not the same as the "sender" and "recipient". The sender and recipient are part of SMTP, the To and From lines are part of the message data and are completely unrelated. Think of it this way: when a letter is sent through the post office, the name on the outside of the envelope tells the postman which mailbox gets the envelope (or where to send it back to) but top of the letter inside may have a completely unrelated letterhead and salutation. Whenever spamdyke's options/documentation refer to a "sender" or a "recipient", it means the name on the outside of the envelope. The user never sees those values in their mail client unless the sender chooses to use those values in the To and From fields. Spammers typically fake all sender/recipient/To/From fields, but other software does too for perfectly legitimate reasons (e.g. mailing lists, autoresponders). Right. If you want to block based on the To and From lines the user sees in their mail
Re: [spamdyke-users] recipient-blacklist-file=FILE with RegExes?
People, I thought of starting a new thread but the question relates to this discussion so I thought I would revive it - see inline comments: On 2015-06-21 04:57, Philip Rhoades via spamdyke-users wrote: Sam, On 2015-06-21 03:12, Sam Clippinger via spamdyke-users wrote: Regex support is on the (rather lengthy) to-do list, but frankly it's not a very high priority -- there's a lot of low-hanging fruit that would be of much more benefit right now. Plus, since I'm not one of the 10 people in the world who completely understands regexes, I doubt I would actually use them myself; I'd rather add globbing support, which I do understand. :) OK, no worries - SD is going well so far so I may not need some of the mechanisms that I used in my own setup - we'll see how things go. spamdyke's header filter runs at connection time, as all of its filters do. If a header line matches a blacklisted pattern, the entire message is rejected (the sending server receives an error code, qmail never sees the message). Right - thanks for the clarification. One annoying spammer continues to get their mail through but I don't understand why - my header-blacklist-file includes these two lines in it: [FR][re][op][ml]*:*brewster.com* [FR][re][op][ml]*:*nice.com* but the first one works and the second one doesn't!: /var/log/maillog-20151230:Dec 29 17:08:43 prix spamdyke[15684]: DENIED_HEADER_BLACKLISTED from: smartdel...@brewster.com to: p...@pricom.com.au origin_ip: 23.253.183.234 origin_rdns: mail-183-234.mailgun.info auth: (unknown) encryption: (none) reason: /usr/local/bin/srejector2/spamdyke_blacklist_header.txt:11 /var/log/maillog-20151230:Dec 29 17:08:00 prix spamdyke[15609]: ALLOWED from: support.a...@nice.com to: mailer-dae...@pricom.com.au origin_ip: 192.114.148.4 origin_rdns: mailil.nice.com auth: (unknown) encryption: (none) reason: 250_ok_1451369280_qp_15628 I have even saved the file in vim a couple of times and restarted qmail a couple of times but no change in the behaviour - what could the explanation be? Thanks, Phil. On Jun 19, 2015, at 9:09 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: Sam, See inline comments: On 2015-06-20 11:53, Sam Clippinger via spamdyke-users wrote: You're correct spamdyke does not support regexes for any of its options, but you can use a wildcard in a sender or recipient white/blacklist file to match entire domains by prefixing the line with an @ symbol. For example: @example.com [1] [1] Yep, saw that - is it possible to support regexes in the future? Full documentation here: http://www.spamdyke.org/documentation/README.html#REJECTING_RECIPIENTS [2] [2] BUT! Be careful -- the "To" and "From" lines in the message header are not the same as the "sender" and "recipient". The sender and recipient are part of SMTP, the To and From lines are part of the message data and are completely unrelated. Think of it this way: when a letter is sent through the post office, the name on the outside of the envelope tells the postman which mailbox gets the envelope (or where to send it back to) but top of the letter inside may have a completely unrelated letterhead and salutation. Whenever spamdyke's options/documentation refer to a "sender" or a "recipient", it means the name on the outside of the envelope. The user never sees those values in their mail client unless the sender chooses to use those values in the To and From fields. Spammers typically fake all sender/recipient/To/From fields, but other software does too for perfectly legitimate reasons (e.g. mailing lists, autoresponders). Right. If you want to block based on the To and From lines the user sees in their mail client, you should look at spamdyke's header blacklist filter: http://www.spamdyke.org/documentation/README.html#HEADERS [3] [3] In that case the mail has already been accepted? When I was using the qmail-qfilter+Ruby script method - my understanding of it at least - was that my Ruby script could process the header and body of the email and exit with a particular error code if the mail was bad and this would terminate the SMTP negotiation with that error message (eg drop the mail silently). So in this case I was able to look at all the header fields as well as the mail body and do whatever I wanted before accepting the mail. Header filtering doesn't support regexes either, but it does use "globbing" to allow more wildcard options. Right. Thanks, Phil. On Jun 19, 2015, at 7:47 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: People, As well as using GreyLite I have done my own thing for many years with qmail-qfilter and a Ruby script (it started off as a Ruby learning exercise . . ) - anyway for my white and black lists I was able to have in the plain text files things like: ad...@phillipsfinancial.com.au administrator@(boo
Re: [spamdyke-users] Blocking "Reply-To:" addresses
Sam, On 2015-10-12 09:45, Sam Clippinger via spamdyke-users wrote: I'm not sure I understand your question. If you want to block messages without a "From" line in their header, spamdyke can't do that. You may be able to use a secondary filter like maildrop to delete the message after it is accepted however. The original problem was that the "From:" header might have something that was believable but the "Reply-to:" header was always dodgy - (re)learning about the difference between the SMTP envelope and mail header stuff clarified things in my own head and finding out about how the header-blacklist-file works essentially solved all of my problems relating to this thread. What I have now blocks anyone I don't like in either the "From:" or "Reply-to:" fields - so I am happy! After a decent amount of time I will post updated stats so we can see how much more spam is being stopped over the basic setup - it won't be much but it will be interesting . . Regards, Phil. -- Sam Clippinger On Oct 9, 2015, at 10:17 AM, Linux via spamdyke-users <spamdyke-users@spamdyke.org> wrote: sorry to hang me for this post, but I would consult them taking advantage of the conversation can be locked via e-mail comes without sender? I'm getting a lot of spam that has this pattern. Best regards, Paul 2015-10-03 1:05 GMT-03:00 Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org>: Sam, On 2015-10-02 23:47, Sam Clippinger via spamdyke-users wrote: I guess so, but remember the wildcarding uses globbing, not regexes. What I mean is: using "?*" is equivalent to just "*". Right. Also, the line has to contain at least one colon or spamdyke won't use it (message headers always use a colon to separate the field name from the value). Yep. Why not just use multiple entries in the file? If either one matches, the message will be blocked and it'd be easier to understand: From: *@skysoft.com [1] [1] Reply-To: *@skysoft.com [1] [1] Doubling the number of lines offends my sensibilities . . this works: [FR][re][op][ml]*:*iskysoft.com [2]* Also, sorting this issue out forced me to sort out the rDNS problem for my main web server - so thanks for that too! Regards, Phil. -- Sam Clippinger On Oct 2, 2015, at 4:34 AM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: On 2015-10-02 15:42, Philip Rhoades via spamdyke-users wrote: Sam, On 2015-09-26 01:12, Sam Clippinger via spamdyke-users wrote: The header blacklist file has a different format from the sender blacklist file, so just copying entries from one to the other won't work. You need to provide a pattern that matches the line(s) in the message header -- in your mail client, you should have an option to "view message source" or "view raw headers" that will show you what it looks like. In this specific case, you probably want this: Reply-To: *@skysoft.com [1] [1] [3]* The format is case insensitive and uses globbing for wildcards, so * will match multiple characters and [] will match a set or range of characters, just like the bash command prompt. The filter will ignore any lines in the file that don't contain a colon. Full details here: http://www.spamdyke.org/documentation/README.html#HEADERS [3] [2] [4] So if I wanted to block the same address for both From: and Reply-To: I could use: [fr][re][op][ml].*@skysoft.com [1] [1] [fr][re][op][ml]?*@skysoft.com [1] [1] so "*" doesn't repeat only "[ml]" ? ? Thanks, Phil. For testing, you certainly can use telnet -- I do it all the time. Just make sure the host you telnet from isn't blocked or whitelisted for some other reason (most folks whitelist localhost, for example). -- Sam Clippinger On Sep 25, 2015, at 1:31 AM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: Sam, On 2015-09-15 07:27, Sam Clippinger via spamdyke-users wrote: Actually, no. The sender-blacklist-* and recipient-blacklist-* filters operate on different data from the header-blacklist-* filters. The reason is because the sender and recipient addresses are given during the SMTP protocol and aren't part of the message itself -- the addresses you see in your mail client are the From and To entries from the message header. The first paragraph here explains in a little more detail: http://www.spamdyke.org/documentation/README.html#REJECTING_SENDERS [4] [3] [1] [1] Yes, sorry, I should have realised that . . Put another way, the sender address doesn't have to match the "From" address visible in the mail client -- well-behaved mail clients make them the same, but that's a courtesy and not a requirement. The Reply-To address is part of the message header and, again, is only a convention used by well-behaved clients. If you've ever been Bcc'd on a message, you've seen this in action -- the sender's mail client gave your address as a recipient but didn
Re: [spamdyke-users] Blocking "Reply-To:" addresses
On 2015-10-02 15:42, Philip Rhoades via spamdyke-users wrote: Sam, On 2015-09-26 01:12, Sam Clippinger via spamdyke-users wrote: The header blacklist file has a different format from the sender blacklist file, so just copying entries from one to the other won't work. You need to provide a pattern that matches the line(s) in the message header -- in your mail client, you should have an option to "view message source" or "view raw headers" that will show you what it looks like. In this specific case, you probably want this: Reply-To: *@skysoft.com [3]* The format is case insensitive and uses globbing for wildcards, so * will match multiple characters and [] will match a set or range of characters, just like the bash command prompt. The filter will ignore any lines in the file that don't contain a colon. Full details here: http://www.spamdyke.org/documentation/README.html#HEADERS [4] So if I wanted to block the same address for both From: and Reply-To: I could use: [fr][re][op][ml].*@skysoft.com [fr][re][op][ml]?*@skysoft.com so "*" doesn't repeat only "[ml]" ? ? Thanks, Phil. For testing, you certainly can use telnet -- I do it all the time. Just make sure the host you telnet from isn't blocked or whitelisted for some other reason (most folks whitelist localhost, for example). -- Sam Clippinger On Sep 25, 2015, at 1:31 AM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: Sam, On 2015-09-15 07:27, Sam Clippinger via spamdyke-users wrote: Actually, no. The sender-blacklist-* and recipient-blacklist-* filters operate on different data from the header-blacklist-* filters. The reason is because the sender and recipient addresses are given during the SMTP protocol and aren't part of the message itself -- the addresses you see in your mail client are the From and To entries from the message header. The first paragraph here explains in a little more detail: http://www.spamdyke.org/documentation/README.html#REJECTING_SENDERS [1] [1] Yes, sorry, I should have realised that . . Put another way, the sender address doesn't have to match the "From" address visible in the mail client -- well-behaved mail clients make them the same, but that's a courtesy and not a requirement. The Reply-To address is part of the message header and, again, is only a convention used by well-behaved clients. If you've ever been Bcc'd on a message, you've seen this in action -- the sender's mail client gave your address as a recipient but didn't put your address on the "To" line in the message header. Right, so, some follow up questions: I moved the following from the sender-blacklist to the header-blacklist: @iskysoft.com [2] - first in the conf file then later into a separate header-blacklist-file with all the massaged addresses from my old setup - but the sender above still seems to be getting through. I thought the "@" was supposed to act like a wild card? Am I still doing something wrong? When I add addresses etc to blacklists etc, is there any way of doing a test myself to see that the block is working? Using a telnet to port 25 on my qmail server and manually pasting header lines is not a real test is it? Thanks, Phil. -- Sam Clippinger On Sep 13, 2015, at 9:20 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: Sam, On 2015-09-14 11:38, Sam Clippinger via spamdyke-users wrote: I'm not entirely sure I understand your question... if the Reply-To address is always the same, you should be able to block it using the header blacklist filter. Ah . . OK - I will try that but doesn't that mean that: sender-blacklist-entry is redundant - ie: header-blacklist-entry should cover everything? Thanks, Phil. If you're wanting to compare the Reply-To address to the From address or the sender address, spamdyke doesn't have that ability. -- Sam Clippinger On Sep 13, 2015, at 4:11 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: People, One variety of spam that is successfully delivered to me has a different "From:" addresses but the same "Reply-To:" address - I can't see a way of blocking these mails in the conf file via the "Reply-To:" address - is it possible? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org
Re: [spamdyke-users] Blocking "Reply-To:" addresses
Sam, On 2015-10-02 23:47, Sam Clippinger via spamdyke-users wrote: I guess so, but remember the wildcarding uses globbing, not regexes. What I mean is: using "?*" is equivalent to just "*". Right. Also, the line has to contain at least one colon or spamdyke won't use it (message headers always use a colon to separate the field name from the value). Yep. Why not just use multiple entries in the file? If either one matches, the message will be blocked and it'd be easier to understand: From: *@skysoft.com [1] Reply-To: *@skysoft.com [1] Doubling the number of lines offends my sensibilities . . this works: [FR][re][op][ml]*:*iskysoft.com* Also, sorting this issue out forced me to sort out the rDNS problem for my main web server - so thanks for that too! Regards, Phil. -- Sam Clippinger On Oct 2, 2015, at 4:34 AM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: On 2015-10-02 15:42, Philip Rhoades via spamdyke-users wrote: Sam, On 2015-09-26 01:12, Sam Clippinger via spamdyke-users wrote: The header blacklist file has a different format from the sender blacklist file, so just copying entries from one to the other won't work. You need to provide a pattern that matches the line(s) in the message header -- in your mail client, you should have an option to "view message source" or "view raw headers" that will show you what it looks like. In this specific case, you probably want this: Reply-To: *@skysoft.com [1] [3]* The format is case insensitive and uses globbing for wildcards, so * will match multiple characters and [] will match a set or range of characters, just like the bash command prompt. The filter will ignore any lines in the file that don't contain a colon. Full details here: http://www.spamdyke.org/documentation/README.html#HEADERS [2] [4] So if I wanted to block the same address for both From: and Reply-To: I could use: [fr][re][op][ml].*@skysoft.com [1] [fr][re][op][ml]?*@skysoft.com [1] so "*" doesn't repeat only "[ml]" ? ? Thanks, Phil. For testing, you certainly can use telnet -- I do it all the time. Just make sure the host you telnet from isn't blocked or whitelisted for some other reason (most folks whitelist localhost, for example). -- Sam Clippinger On Sep 25, 2015, at 1:31 AM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: Sam, On 2015-09-15 07:27, Sam Clippinger via spamdyke-users wrote: Actually, no. The sender-blacklist-* and recipient-blacklist-* filters operate on different data from the header-blacklist-* filters. The reason is because the sender and recipient addresses are given during the SMTP protocol and aren't part of the message itself -- the addresses you see in your mail client are the From and To entries from the message header. The first paragraph here explains in a little more detail: http://www.spamdyke.org/documentation/README.html#REJECTING_SENDERS [3] [1] [1] Yes, sorry, I should have realised that . . Put another way, the sender address doesn't have to match the "From" address visible in the mail client -- well-behaved mail clients make them the same, but that's a courtesy and not a requirement. The Reply-To address is part of the message header and, again, is only a convention used by well-behaved clients. If you've ever been Bcc'd on a message, you've seen this in action -- the sender's mail client gave your address as a recipient but didn't put your address on the "To" line in the message header. Right, so, some follow up questions: I moved the following from the sender-blacklist to the header-blacklist: @iskysoft.com [2] - first in the conf file then later into a separate header-blacklist-file with all the massaged addresses from my old setup - but the sender above still seems to be getting through. I thought the "@" was supposed to act like a wild card? Am I still doing something wrong? When I add addresses etc to blacklists etc, is there any way of doing a test myself to see that the block is working? Using a telnet to port 25 on my qmail server and manually pasting header lines is not a real test is it? Thanks, Phil. -- Sam Clippinger On Sep 13, 2015, at 9:20 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: Sam, On 2015-09-14 11:38, Sam Clippinger via spamdyke-users wrote: I'm not entirely sure I understand your question... if the Reply-To address is always the same, you should be able to block it using the header blacklist filter. Ah . . OK - I will try that but doesn't that mean that: sender-blacklist-entry is redundant - ie: header-blacklist-entry should cover everything? Thanks, Phil. If you're wanting to compare the Reply-To address to the From address or the sender address, spamdyke doesn't have that ability. -- Sam Clippinger On Sep 13, 2015, at 4:11 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: Peop
Re: [spamdyke-users] Blocking "Reply-To:" addresses
Sam, On 2015-09-26 01:12, Sam Clippinger via spamdyke-users wrote: The header blacklist file has a different format from the sender blacklist file, so just copying entries from one to the other won't work. You need to provide a pattern that matches the line(s) in the message header -- in your mail client, you should have an option to "view message source" or "view raw headers" that will show you what it looks like. In this specific case, you probably want this: Reply-To: *@skysoft.com [3]* The format is case insensitive and uses globbing for wildcards, so * will match multiple characters and [] will match a set or range of characters, just like the bash command prompt. The filter will ignore any lines in the file that don't contain a colon. Full details here: http://www.spamdyke.org/documentation/README.html#HEADERS [4] So if I wanted to block the same address for both From: and Reply-To: I could use: [fr][re][op][ml].*@skysoft.com ? Thanks, Phil. For testing, you certainly can use telnet -- I do it all the time. Just make sure the host you telnet from isn't blocked or whitelisted for some other reason (most folks whitelist localhost, for example). -- Sam Clippinger On Sep 25, 2015, at 1:31 AM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: Sam, On 2015-09-15 07:27, Sam Clippinger via spamdyke-users wrote: Actually, no. The sender-blacklist-* and recipient-blacklist-* filters operate on different data from the header-blacklist-* filters. The reason is because the sender and recipient addresses are given during the SMTP protocol and aren't part of the message itself -- the addresses you see in your mail client are the From and To entries from the message header. The first paragraph here explains in a little more detail: http://www.spamdyke.org/documentation/README.html#REJECTING_SENDERS [1] [1] Yes, sorry, I should have realised that . . Put another way, the sender address doesn't have to match the "From" address visible in the mail client -- well-behaved mail clients make them the same, but that's a courtesy and not a requirement. The Reply-To address is part of the message header and, again, is only a convention used by well-behaved clients. If you've ever been Bcc'd on a message, you've seen this in action -- the sender's mail client gave your address as a recipient but didn't put your address on the "To" line in the message header. Right, so, some follow up questions: I moved the following from the sender-blacklist to the header-blacklist: @iskysoft.com [2] - first in the conf file then later into a separate header-blacklist-file with all the massaged addresses from my old setup - but the sender above still seems to be getting through. I thought the "@" was supposed to act like a wild card? Am I still doing something wrong? When I add addresses etc to blacklists etc, is there any way of doing a test myself to see that the block is working? Using a telnet to port 25 on my qmail server and manually pasting header lines is not a real test is it? Thanks, Phil. -- Sam Clippinger On Sep 13, 2015, at 9:20 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: Sam, On 2015-09-14 11:38, Sam Clippinger via spamdyke-users wrote: I'm not entirely sure I understand your question... if the Reply-To address is always the same, you should be able to block it using the header blacklist filter. Ah . . OK - I will try that but doesn't that mean that: sender-blacklist-entry is redundant - ie: header-blacklist-entry should cover everything? Thanks, Phil. If you're wanting to compare the Reply-To address to the From address or the sender address, spamdyke doesn't have that ability. -- Sam Clippinger On Sep 13, 2015, at 4:11 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: People, One variety of spam that is successfully delivered to me has a different "From:" addresses but the same "Reply-To:" address - I can't see a way of blocking these mails in the conf file via the "Reply-To:" address - is it possible? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users Links: -- [1] http://www.spamdyke.org/documentation/README.html#REJECTING_SENDERS ___ spamdyke-users mailing list spam
Re: [spamdyke-users] Blocking "Reply-To:" addresses
Martin, On 2015-09-26 22:10, Martin H. Sluka via spamdyke-users wrote: Sam wrote: For testing, you certainly can use telnet -- I do it all the time. Tip: You might want to have a look at Swaks (Swiss Army Knife for SMTP, http://www.jetmore.org/john/code/swaks/). I find it very convenient for testing and monitoring purposes, especially if you want to perform similar tests several times. Thanks for the reminder! I had forgotten about swaks . . Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Blocking "Reply-To:" addresses
Sam, On 2015-09-15 07:27, Sam Clippinger via spamdyke-users wrote: Actually, no. The sender-blacklist-* and recipient-blacklist-* filters operate on different data from the header-blacklist-* filters. The reason is because the sender and recipient addresses are given during the SMTP protocol and aren't part of the message itself -- the addresses you see in your mail client are the From and To entries from the message header. The first paragraph here explains in a little more detail: http://www.spamdyke.org/documentation/README.html#REJECTING_SENDERS [1] Yes, sorry, I should have realised that . . Put another way, the sender address doesn't have to match the "From" address visible in the mail client -- well-behaved mail clients make them the same, but that's a courtesy and not a requirement. The Reply-To address is part of the message header and, again, is only a convention used by well-behaved clients. If you've ever been Bcc'd on a message, you've seen this in action -- the sender's mail client gave your address as a recipient but didn't put your address on the "To" line in the message header. Right, so, some follow up questions: I moved the following from the sender-blacklist to the header-blacklist: @iskysoft.com - first in the conf file then later into a separate header-blacklist-file with all the massaged addresses from my old setup - but the sender above still seems to be getting through. I thought the "@" was supposed to act like a wild card? Am I still doing something wrong? When I add addresses etc to blacklists etc, is there any way of doing a test myself to see that the block is working? Using a telnet to port 25 on my qmail server and manually pasting header lines is not a real test is it? Thanks, Phil. -- Sam Clippinger On Sep 13, 2015, at 9:20 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: Sam, On 2015-09-14 11:38, Sam Clippinger via spamdyke-users wrote: I'm not entirely sure I understand your question... if the Reply-To address is always the same, you should be able to block it using the header blacklist filter. Ah . . OK - I will try that but doesn't that mean that: sender-blacklist-entry is redundant - ie: header-blacklist-entry should cover everything? Thanks, Phil. If you're wanting to compare the Reply-To address to the From address or the sender address, spamdyke doesn't have that ability. -- Sam Clippinger On Sep 13, 2015, at 4:11 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: People, One variety of spam that is successfully delivered to me has a different "From:" addresses but the same "Reply-To:" address - I can't see a way of blocking these mails in the conf file via the "Reply-To:" address - is it possible? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users Links: -- [1] http://www.spamdyke.org/documentation/README.html#REJECTING_SENDERS ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] Blocking "Reply-To:" addresses
People, One variety of spam that is successfully delivered to me has a different "From:" addresses but the same "Reply-To:" address - I can't see a way of blocking these mails in the conf file via the "Reply-To:" address - is it possible? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Blocking "Reply-To:" addresses
Sam, On 2015-09-14 11:38, Sam Clippinger via spamdyke-users wrote: I'm not entirely sure I understand your question... if the Reply-To address is always the same, you should be able to block it using the header blacklist filter. Ah . . OK - I will try that but doesn't that mean that: sender-blacklist-entry is redundant - ie: header-blacklist-entry should cover everything? Thanks, Phil. If you're wanting to compare the Reply-To address to the From address or the sender address, spamdyke doesn't have that ability. -- Sam Clippinger On Sep 13, 2015, at 4:11 PM, Philip Rhoades via spamdyke-users <spamdyke-users@spamdyke.org> wrote: People, One variety of spam that is successfully delivered to me has a different "From:" addresses but the same "Reply-To:" address - I can't see a way of blocking these mails in the conf file via the "Reply-To:" address - is it possible? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] Some stats after a couple of months; NotInFromWhiteList; Calling External Program
People, Here are some stats after a couple of months of happy Spamdyke usage - thanks! If I had remembered to set the logrotate number higher I would have had more data but I think the last 31 days is sufficient to illustrate some things: Total spamdyke lines in maillog files for the last 31 days: 54838 Total spamdyke ALLOWED lines in maillog files for the last 31 days: 12278 (22.4%) Total spam / phishing messages that were delivered: 165 100% Valid To email address: 105 64% No To email address: 19 12% Undisclosed Recipients: 159% Mailer Daemon bounces:138% Invalid To email address: 127% Valid To email address but NO Subject and NO From: 11% I could stop the 64% Valid To email address spams if I had a NotInFromWhiteList facility - at the expense of annoying people sometimes with failed messages and them receiving a If you are a real mailer . . note - like my previous Qmail + GreyLite + Ruby script (that was called via qmail-qfilter) setup. Except for Mailer Daemon bounces ands Valid To email address but NO SUBJECT and NO FROM, I don't even know how the other mails actually get delivered at all . . I notice the processing that spamdyke does is slower for me to send mail compared to my previous setup - but I guess it is doing more work too . . Is there any way for me to call a modified version of my old Ruby script from spamdyke as the last bit of processing before allowing an email through? Thanks again! Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Softlimit messages
Sam, OK, I am convinced - will delete . . Thanks, Phil. On 2015-06-21 09:12, Sam Clippinger via spamdyke-users wrote: IMHO, everyone should delete the softlimit program from their servers immediately. Not that I have a strong opinion on the matter or anything. :) The softlimit program seems like a good idea -- set an upper limit on the amount of RAM a program can use, to guard against memory leaks (but not buffer overflows). In practice however, it causes far far more problems than it causes. When a program hits the limit, it always happens inside a library function and not the application itself. So the user sees strange errors from glibc or OpenSSL functions that are never related to memory allocation. Those errors always look like real bugs, because there's never any indication the limit was hit. There's also no way to even estimate how much memory is correct. Does anyone really understand how many libraries a program loads and how much memory they need? spamdyke uses OpenSSL and on some systems, separate libraries for math and DNS functions. Unpatched qmail doesn't use many libraries, but if patches have been applied to allow TLS or authentication, it may use many (who uses unpatched qmail anyway?). If vpopmail is in use, it may need MySQL, depending on how it was compiled. If the server is configured to use stack guarding or memory profiling, the virtual memory use could be astronomical. Every guide I've ever read says to use trial-and-error to find the lowest value that appears to work, then double (or triple) it. Crazy! I've spent way way too much time trying to track down bugs that were caused by softlimit and I finally reached my own limit this year. That's why spamdyke 5.0.1 examines the limits it starts with and, if it can, resets them. It can't undo hard limits set by the ulimit program, but it can (and does) undo softlimit. -- Sam Clippinger On Jun 20, 2015, at 2:05 PM, Philip Rhoades via spamdyke-users spamdyke-users@spamdyke.org wrote: People, I played around with the logging verbosity and found if I used debug mode I saw suggestions (commands!?) in the log about remove the softlimit function from the start script for qmail-smtpd - while I was trying to sort out the last bug that was preventing eQmail from working, I did actually do that - is the softlimit function even necessary these days on a lightly loaded server with 8GB RAM? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] Softlimit messages
People, I played around with the logging verbosity and found if I used debug mode I saw suggestions (commands!?) in the log about remove the softlimit function from the start script for qmail-smtpd - while I was trying to sort out the last bug that was preventing eQmail from working, I did actually do that - is the softlimit function even necessary these days on a lightly loaded server with 8GB RAM? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] recipient-blacklist-file=FILE with RegExes?
Sam, On 2015-06-21 03:12, Sam Clippinger via spamdyke-users wrote: Regex support is on the (rather lengthy) to-do list, but frankly it's not a very high priority -- there's a lot of low-hanging fruit that would be of much more benefit right now. Plus, since I'm not one of the 10 people in the world who completely understands regexes, I doubt I would actually use them myself; I'd rather add globbing support, which I do understand. :) OK, no worries - SD is going well so far so I may not need some of the mechanisms that I used in my own setup - we'll see how things go. spamdyke's header filter runs at connection time, as all of its filters do. If a header line matches a blacklisted pattern, the entire message is rejected (the sending server receives an error code, qmail never sees the message). Right - thanks for the clarification. Regards, Phil. -- Sam Clippinger On Jun 19, 2015, at 9:09 PM, Philip Rhoades via spamdyke-users spamdyke-users@spamdyke.org wrote: Sam, See inline comments: On 2015-06-20 11:53, Sam Clippinger via spamdyke-users wrote: You're correct spamdyke does not support regexes for any of its options, but you can use a wildcard in a sender or recipient white/blacklist file to match entire domains by prefixing the line with an @ symbol. For example: @example.com [1] [1] Yep, saw that - is it possible to support regexes in the future? Full documentation here: http://www.spamdyke.org/documentation/README.html#REJECTING_RECIPIENTS [2] [2] BUT! Be careful -- the To and From lines in the message header are not the same as the sender and recipient. The sender and recipient are part of SMTP, the To and From lines are part of the message data and are completely unrelated. Think of it this way: when a letter is sent through the post office, the name on the outside of the envelope tells the postman which mailbox gets the envelope (or where to send it back to) but top of the letter inside may have a completely unrelated letterhead and salutation. Whenever spamdyke's options/documentation refer to a sender or a recipient, it means the name on the outside of the envelope. The user never sees those values in their mail client unless the sender chooses to use those values in the To and From fields. Spammers typically fake all sender/recipient/To/From fields, but other software does too for perfectly legitimate reasons (e.g. mailing lists, autoresponders). Right. If you want to block based on the To and From lines the user sees in their mail client, you should look at spamdyke's header blacklist filter: http://www.spamdyke.org/documentation/README.html#HEADERS [3] [3] In that case the mail has already been accepted? When I was using the qmail-qfilter+Ruby script method - my understanding of it at least - was that my Ruby script could process the header and body of the email and exit with a particular error code if the mail was bad and this would terminate the SMTP negotiation with that error message (eg drop the mail silently). So in this case I was able to look at all the header fields as well as the mail body and do whatever I wanted before accepting the mail. Header filtering doesn't support regexes either, but it does use globbing to allow more wildcard options. Right. Thanks, Phil. On Jun 19, 2015, at 7:47 PM, Philip Rhoades via spamdyke-users spamdyke-users@spamdyke.org wrote: People, As well as using GreyLite I have done my own thing for many years with qmail-qfilter and a Ruby script (it started off as a Ruby learning exercise . . ) - anyway for my white and black lists I was able to have in the plain text files things like: ad...@phillipsfinancial.com.au administrator@(booksjournals.com [4](|.au)|(prix.|)pricom.com.au [5]|qps.com.au [6]) adwords-noreply america.com [7] ecolife where if any of those particular regexes appeared in the To: or From: or whatever, they could be allowed or blocked or whatever - I am guessing that eg the recipient-blacklist-file=FILE only allows for full email addresses? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users Links: -- [1] http://example.com [1] [2] http://www.spamdyke.org/documentation/README.html#REJECTING_RECIPIENTS [2] [3] http://www.spamdyke.org/documentation/README.html#HEADERS [3] ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users Links: -- [1] http://example.com [2] http://www.spamdyke.org/documentation/README.html#REJECTING_RECIPIENTS [3
Re: [spamdyke-users] Moving from GreyLite
Gary and Sam, Thanks for the useful info! I have SpamDyke running now with the simple conf and will start looking at the options. I have some white black lists to import to . . BTW, it appears top-posting is OK here? Regards, Phil. On 2015-06-20 05:52, Sam Clippinger via spamdyke-users wrote: I'm not familiar with GreyLite at all, but connection-time means spamdyke does its work while the message is still coming into your mail server -- while the connection with the sending server is active. This is as opposed to filtering messages in the mail queue, after the remote server is no longer connected (and believes the message has been delivered). The advantage of a connection-time filter is the remote server sees the rejection and the spam is never stored on your server at all. Rejecting messages after they've been queued requires either sending a bounce message or delivering it to a user's Junk folder. This distinction comes up a lot around qmail regarding recipient validation. By itself, qmail does not validate recipients when messages are accepted. Any username at a valid domain is accepted, then bounced later if the address turns out to be invalid. This leads to the problem of backscatter spam -- spammers deliberately send messages to invalid addresses and set the from address to their intended target. A qmail server will bounce the message (complete with spam or virus) to the victim. For qmail to validate recipients at connection time requires a patch or a filter like spamdyke. -- Sam Clippinger On Jun 19, 2015, at 5:21 AM, Philip Rhoades via spamdyke-users spamdyke-users@spamdyke.org wrote: People, I have been using GreyLite for many years but it hasn't been supported for quite a while - I think it is time to update to SpamDyke . . but I have some questions - first one: I looked at the SpamDyke web site and it is still not clear to me - it says 'connection-time means spamdyke evaluates and rejects spam while the remote server is still delivering it' - does this mean it does it at the TCP / mail envelope level? ie so it would be the same as GreyLite? GL blocks and forces possibly bad mails to be resent some time later which many spammers don't attempt . . Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] recipient-blacklist-file=FILE with RegExes?
People, As well as using GreyLite I have done my own thing for many years with qmail-qfilter and a Ruby script (it started off as a Ruby learning exercise . . ) - anyway for my white and black lists I was able to have in the plain text files things like: ad...@phillipsfinancial.com.au administrator@(booksjournals.com(|.au)|(prix.|)pricom.com.au|qps.com.au) adwords-noreply america.com ecolife where if any of those particular regexes appeared in the To: or From: or whatever, they could be allowed or blocked or whatever - I am guessing that eg the recipient-blacklist-file=FILE only allows for full email addresses? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] recipient-blacklist-file=FILE with RegExes?
Sam, See inline comments: On 2015-06-20 11:53, Sam Clippinger via spamdyke-users wrote: You're correct spamdyke does not support regexes for any of its options, but you can use a wildcard in a sender or recipient white/blacklist file to match entire domains by prefixing the line with an @ symbol. For example: @example.com [1] Yep, saw that - is it possible to support regexes in the future? Full documentation here: http://www.spamdyke.org/documentation/README.html#REJECTING_RECIPIENTS [2] BUT! Be careful -- the To and From lines in the message header are not the same as the sender and recipient. The sender and recipient are part of SMTP, the To and From lines are part of the message data and are completely unrelated. Think of it this way: when a letter is sent through the post office, the name on the outside of the envelope tells the postman which mailbox gets the envelope (or where to send it back to) but top of the letter inside may have a completely unrelated letterhead and salutation. Whenever spamdyke's options/documentation refer to a sender or a recipient, it means the name on the outside of the envelope. The user never sees those values in their mail client unless the sender chooses to use those values in the To and From fields. Spammers typically fake all sender/recipient/To/From fields, but other software does too for perfectly legitimate reasons (e.g. mailing lists, autoresponders). Right. If you want to block based on the To and From lines the user sees in their mail client, you should look at spamdyke's header blacklist filter: http://www.spamdyke.org/documentation/README.html#HEADERS [3] In that case the mail has already been accepted? When I was using the qmail-qfilter+Ruby script method - my understanding of it at least - was that my Ruby script could process the header and body of the email and exit with a particular error code if the mail was bad and this would terminate the SMTP negotiation with that error message (eg drop the mail silently). So in this case I was able to look at all the header fields as well as the mail body and do whatever I wanted before accepting the mail. Header filtering doesn't support regexes either, but it does use globbing to allow more wildcard options. Right. Thanks, Phil. On Jun 19, 2015, at 7:47 PM, Philip Rhoades via spamdyke-users spamdyke-users@spamdyke.org wrote: People, As well as using GreyLite I have done my own thing for many years with qmail-qfilter and a Ruby script (it started off as a Ruby learning exercise . . ) - anyway for my white and black lists I was able to have in the plain text files things like: ad...@phillipsfinancial.com.au administrator@(booksjournals.com(|.au)|(prix.|)pricom.com.au|qps.com.au) adwords-noreply america.com ecolife where if any of those particular regexes appeared in the To: or From: or whatever, they could be allowed or blocked or whatever - I am guessing that eg the recipient-blacklist-file=FILE only allows for full email addresses? Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users Links: -- [1] http://example.com [2] http://www.spamdyke.org/documentation/README.html#REJECTING_RECIPIENTS [3] http://www.spamdyke.org/documentation/README.html#HEADERS ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] Moving from GreyLite
People, I have been using GreyLite for many years but it hasn't been supported for quite a while - I think it is time to update to SpamDyke . . but I have some questions - first one: I looked at the SpamDyke web site and it is still not clear to me - it says 'connection-time means spamdyke evaluates and rejects spam while the remote server is still delivering it' - does this mean it does it at the TCP / mail envelope level? ie so it would be the same as GreyLite? GL blocks and forces possibly bad mails to be resent some time later which many spammers don't attempt . . Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: p...@pricom.com.au ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
