Hi Eric,
After reading you messages it seems to me that the following is what
you are searching for:
This is from one of my dspam installs (postfix - master.cf).
dspam-retrain unix - n n - - pipe flags=XRu user=dspam
argv=/usr/local/bin/dspam-retrain-forward.pl
--debug=no
--client
--user ${sender}
--class=${nexthop}
--source=error
--full=yes
--headers-only=no
--bodies-only=yes
--first-only=no
--skip-first=no
--bin-dir=/usr/bin
Messages forwarded to the retaining addresses nots...@example.com and
s...@example.com are sent to the dspam-retrain transport.
As you can see the user is populated from the message sender address.
The address of the user that sent the message for retraining.
Obviously this has implications for aliases!!! They should only be
expanded post Dspam filter otherwise you will have problems
retraining.
Hopefully that will help, or a may have just miss understood what you
are trying to achieve.
Best Regards
Martin
On 2015-02-15 07:06, Eric Broch wrote:
> "How do I tell the dspam engine that a message,
> identified by signature with uid included,
> without the --user supplied,
> is to be re-learned as spam/innocent?"
>
> I know my implementations are working properly and they won't retrain
> a message without the '--user' parameter. Seems to me it is
> necessary.
> Maybe, an expert will chime in on the issue.
>
> Eric
>
>
> On 2/14/2015 11:37 PM, Jeff Kletsky wrote:
>> OK, I've been digging into the source code as well as observing
>> different behavior when
>> running dspamc as a normal user from what is seen when running as
>> root.
>>
>> First off, it looks as though, even with MySQLUIDInSignature, that
>> you
>> *must* supply the user in the command line, unless you are using
>> ParseToHeader.
>>
>> So I don't lose everyone with the details, the key question is
>>
>> How do I tell the dspam engine that a message,
>> identified by signature with uid included,
>> without the --user supplied,
>> is to be re-learned as spam/innocent?
>>
>>
>> Back to the details:
>>
>> language.h:#define ERR_AGENT_USER_UNDEFINED "Unable to determine
>> the
>> destination user"
>>
>> in check_configuration(AGENT_CTX *ATX):
>>
>> if (!_ds_match_attribute(agent_config, "ParseToHeaders", "on")) {
>>
>> if (ATX->users->items == 0)
>> {
>> LOG(LOG_ERR, ERR_AGENT_USER_UNDEFINED);
>> return EINVAL;
>> }
>> }
>>
>>
>> Now, maybe I'm missing something somewhere, but if you've got the
>> uid in
>> the signature,
>> and you aren't even passing along the message, why is failing to
>> supply
>> an explicit --user
>> an error?
>>
>> Relying on parsing the To: header isn't going to help in many cases,
>> especially as spam is often sent to envelope addresses that aren't
>> listed in the To: header.
>>
>>
>>
>>
>> There also seems, at least at first glance, to be an inconsistency
>> in
>> the code about how Unix-domain sockets are handled. This came up
>> when
>> looking at a non-privileged user getting different error messages
>> than root.
>>
>> jeff@mail:/ % dspamc --source=error --class=spam
>> --signature=2,54deafa8740195493713973
>>
>> Feb 14 20:56:27 mail dspam[80659]: Invalid client configuration
>>
>>
>> Looking into the "Invalid client configuration" gives two possible
>> places it could fail
>>
>> 1) There is a check at the end of dspamc.c, right before calling
>> client_process(), for (ClientIdent and ClientHost) or
>> ServerDomainSocketPath. The latter is has been confirmed set, so
>> that
>> shouldn't be the cause.
>>
>> 2) In client.c in client_connect() the check is slightly different:
>>
>> host = _ds_read_attribute(agent_config, "ClientHost");
>>
>> if (_ds_read_attribute(agent_config, "ClientPort"))
>> port = atoi(_ds_read_attribute(agent_config, "ClientPort"));
>>
>> if (host && host[0] == '/')
>> domain = 1;
>>
>> Note that there is no check for ServerDomainSocketPath here. I
>> haven't
>> confirmed which part of the code path is raising the error. However,
>> making the following changes to dspam.config allows the client to
>> connect (and, foreshadowing, fail differently):
>>
>> -#ServerPass.Relay1 "secret"
>> +ServerPass.Relay1 "secret"
>>
>> -#ClientHost /var/run/dspam.sock
>> -#ClientIdent "secret@Relay1"
>> +ClientHost "/var/spool/postfix/dspam/dspam.sock"
>> +ClientIdent "secret@Relay1"
>>
>>
>> I'm still disappointed that I apparently will have to parse the
>>
>> Now, the log errors are (and only changed for the non-privileged
>> user)
>>
>> Feb 14 21:54:16 mail dspam[83195]: Signature retrieval for
>> '10,54df1c80752941515871655' failed
>> Feb 14 21:54:16 mail dspam[83195]: Unable to find a valid signature.
>> Aborting.
>> Feb 14 21:54:16 mail dspam[83195]: process_message returned error
>> -5.
>> dropping message.
>>
>> These messages could use some improvement as it isn't the signature
>> retrieval that failed, but that there wasn't a user in the virtual
>> user
>> table with the same name as the user executing dspamc
>>
>> /var/log/dspam/dspam.debug:
>>
>> 83195: [02/14/2015 22:04:16] checking trusted user list for root(0)
>> 83195: [02/14/2015 22:04:16] process mode: '--source=error
>> --signature=10,54df1c80752941515871655 --class=spam --debug '
>> 83195: [02/14/2015 22:04:16] No QuarantineAgent option found. Using
>> standard quarantine.
>> 83195: [02/14/2015 22:04:16] using database handle id 4
>> 83195: [02/14/2015 22:04:16] handle locked
>> 83195: [02/14/2015 22:04:16] DSPAM Instance Startup
>> 83195: [02/14/2015 22:04:16] input args: dspam --source=error
>> --signature=10,54df1c80752941515871655 --class=spam --debug
>> 83195: [02/14/2015 22:04:16] pass-thru args:
>> 83195: [02/14/2015 22:04:16] processing user jeff
>> 83195: [02/14/2015 22:04:16] uid = 0, euid = 0, gid = 0, egid = 0
>> 83195: [02/14/2015 22:04:16] loading preferences for user jeff
>> 83195: [02/14/2015 22:04:16] default preferences empty. reverting to
>> dspam.conf preferences.
>> 83195: [02/14/2015 22:04:16] Loading preferences from dspam.conf
>> 83195: [02/14/2015 22:04:16] using /var/db/dspam/opt-in/jeff.dspam
>> as path
>> 83195: [02/14/2015 22:04:16] using
>> /var/db/dspam/opt-out/jeff.nodspam as
>> path
>> 83195: [02/14/2015 22:04:16] sedation level set to: 5
>> 83195: [02/14/2015 22:04:16] _mysql_drv_getpwnam: returning NULL for
>> query on name: jeff
>> 83195: [02/14/2015 22:04:16] _mysql_drv_get_spamtotals: unable to
>> _mysql_drv_getpwnam(jeff)
>> 83195: [02/14/2015 22:04:16] _ds_init_storage: unable to load
>> totals.
>> Using zero values.
>> 83195: [02/14/2015 22:04:16] _mysql_drv_getpwnam: returning NULL for
>> query on name: jeff
>> 83195: [02/14/2015 22:04:16] _ds_get_signature: unable to
>> _mysql_drv_getpwnam(jeff)
>> 83195: [02/14/2015 22:04:16] _mysql_drv_getpwnam: returning NULL for
>> query on name: jeff
>> 83195: [02/14/2015 22:04:16] _mysql_drv_set_spamtotals: unable to
>> _mysql_drv_getpwnam(jeff)
>> 83195: [02/14/2015 22:04:16] DSPAM Instance Shutdown. Exit Code: 0
>> 83195: [02/14/2015 22:04:16] checking trusted user list for root(0)
>>
>>
>> So it looks like dspam is still trying to figure out who the system
>> user
>> "jeff" is, when it has been getting virtual domain names all along.
>>
>>
>> Adding 'Trust jeff' lets me add an explicit --user
>> free...@wagsky.com
>> Omitting the --user freebsd@wagsky.comresults in the same "Unable to
>> determine the destination user" error as with root running the
>> command.
>>
>>
>>
>> How can I have dspam determine the appropriate user from the
>> --signature
>> given?
>>
>>
>>
>> Thanks!
>>
>> Jeff
>>
>>
>>
>>
>> On 2/14/15 4:22 PM, Jeff Kletsky wrote:
>>> First, some thanks to those that have already posted and given me
>>> some
>>> insights along the way in getting
>>>
>>> * dovecot2-2.2.15_3 Secure, fast and powerful IMAP and
>>> POP3
>>> server
>>> * postfix-2.11.3_4,1 Secure alternative to widely-used
>>> Sendmail
>>> * dspam-3.10.2 Bayesian spam filter
>>> * mysql56-server-5.6.22 Multithreaded SQL database
>>> (server)
>>>
>>> up and running (mainly) together, jailed on FreeBSD 9.3
>>>
>>> I've got postfix after-queue filtering the mail through dspam as
>>> per the
>>> "advanced " described at http://www.postfix.org/FILTER_README.html.
>>>
>>> The mail, both innocent and spam, is successfully delivered through
>>> procmail to Maildir and is managed through Dovecot.
>>>
>>> As I have multiple aliases that all get delivered to the same
>>> system
>>> user, I have configured a MySQL back-end with the addition of the
>>> virtual_users.sql schema for dsmap_virtual_uids (the one that has
>>> the
>>> uid generated through auto_increment). This has been enabled in
>>> dspam.conf. (Yes, better error messages than "received invalid
>>> result
>>> (!DSR_ISSPAM && !DSR_ISINNOCENT): -2" would have been helpful.)
>>>
>>>
>>>
>>> I am stumped as to how to debug further though, as when I try to
>>> train
>>> dspam:
>>>
>>> root@mail:/ # dspam --source=error
>>> --signature=10,54df1c80752941515871655 --class=spam --debug
>>> (or)
>>> root@mail:/ # dspamc --source=error
>>> --signature=10,54df1c80752941515871655 --class=spam --debug
>>>
>>> I get
>>>
>>> Feb 14 08:01:58 mail dspam[53601]: Unable to determine the
>>> destination user
>>> Feb 14 08:01:58 mail dspam[53601]: DSPAM agent misconfigured:
>>> aborting
>>>
>>> There aren't any additional clues that I can see in maillog or in
>>> dspam.debug
>>>
>>>
>>> The signature pretty clearly exists in the database, as does the
>>> associated UID
>>>
>>>
>>> mysql> SELECT uid, signature, length, created_on FROM
>>> dspam_signature_data WHERE signature='10,54df1c80752941515871655' ;
>>> +-----+----------------------------+--------+------------+
>>> | uid | signature | length | created_on |
>>> +-----+----------------------------+--------+------------+
>>> | 10 | 10,54df1c80752941515871655 | 37216 | 2015-02-14 |
>>> +-----+----------------------------+--------+------------+
>>> 1 row in set (0.00 sec)
>>>
>>> mysql> SELECT * FROM dspam_virtual_uids WHERE uid='10' ;
>>> +-----+--------------------+
>>> | uid | username |
>>> +-----+--------------------+
>>> | 10 | free...@wagsky.com |
>>> +-----+--------------------+
>>> 1 row in set (0.00 sec)
>>>
>>>
>>> (That email address is from before the days of spam, so it already
>>> gets
>>> hit pretty hard, and I'm not "exposing" it any more here).
>>>
>>> Other signatures for other uids behave the same way.
>>>
>>>
>>>
>>> Any suggestions on where to explore from here to resolve this?
>>>
>>> Thanks,
>>>
>>> Jeff
>>>
>>>
>>> (Needless to say, I'm rapidly building up a lot of spam I can use
>>> for
>>> training!)
>>>
>>>
>>>
>>>
>>> root@mail:/ # dspam --version
>>>
>>> DSPAM Anti-Spam Suite 3.10.2 (agent/library)
>>>
>>> Copyright (C) 2002-2012 DSPAM Project
>>> http://dspam.sourceforge.net.
>>>
>>> DSPAM may be copied only under the terms of the GNU Affero General
>>> Public
>>> License, a copy of which can be found with the DSPAM distribution
>>> kit.
>>>
>>> Configuration parameters: '--sysconfdir=/usr/local/etc'
>>> '--with-logdir=/var/log/dspam' '--with-dspam-home=/var/db/dspam'
>>> '--with-dspam-home-owner=root' '--with-dspam-home-group=mail'
>>> '--with-dspam-home-mode=0770' '--with-dspam-owner=root'
>>> '--with-dspam-group=mail' '--enable-syslog' '--enable-debug'
>>> '--enable-daemon' '--enable-clamav'
>>> '--with-sqlite-includes=/usr/local/include'
>>> '--with-sqlite-libraries=/usr/local/lib'
>>> '--with-mysql-includes=/usr/local/include/mysql'
>>> '--with-mysql-libraries=/usr/local/lib/mysql'
>>> '--enable-client-compression'
>>> '--with-storage-driver=sqlite3_drv,hash_drv,mysql_drv'
>>> '--enable-virtual-users' '--with-dspam-mode=4510'
>>> '--prefix=/usr/local'
>>> '--mandir=/usr/local/man' '--infodir=/usr/local/info/'
>>> '--build=amd64-portbld-freebsd9.2'
>>> 'build_alias=amd64-portbld-freebsd9.2' 'CC=cc' 'CFLAGS=-pipe -g
>>> -fstack-protector -fno-strict-aliasing' 'LDFLAGS= -L/usr/local/lib
>>> -fstack-protector' 'LIBS=' 'CPPFLAGS=-I/usr/local/include'
>>> 'CPP=cpp'
>>>
>>>
>>>
>>>
>>> root@mail:/ # egrep -v '^#' /usr/local/etc/postfix/master.cf |
>>> egrep '^.+$'
>>> smtp inet n - n - - smtpd
>>> -o content_filter=lmtp:unix:dspam/dspam.sock
>>> -o receive_override_options=no_address_mappings
>>> localhost:24 inet n - n - 10 smtpd
>>> -o syslog_name=postfix/reinject
>>> -o content_filter=
>>> -o
>>>
>>> receive_override_options=no_unknown_recipient_checks,no_header_body_checks,no_milters
>>> -o smtpd_helo_restrictions=
>>> -o smtpd_client_restrictions=
>>> -o smtpd_sender_restrictions=
>>> # Postfix 2.10 and later: specify empty
>>> smtpd_relay_restrictions.
>>> -o smtpd_relay_restrictions=
>>> -o smtpd_recipient_restrictions=permit_mynetworks,reject
>>> -o mynetworks=127.0.0.0/8
>>> -o smtpd_authorized_xforward_hosts=127.0.0.0/8
>>> [...]
>>>
>>>
>>>
>>> root@mail:/ # egrep -v '^#' /usr/local/etc/dspam.conf | egrep
>>> '^.+$'
>>> Home /var/db/dspam
>>> StorageDriver /usr/local/lib/dspam/libmysql_drv.so
>>> DeliveryHost 127.0.0.1
>>> DeliveryPort 24
>>> DeliveryIdent localhost
>>> DeliveryProto SMTP
>>> OnFail error
>>> Trust root
>>> Trust dspam
>>> TrainingMode teft
>>> TestConditionalTraining on
>>> Feature whitelist
>>> Feature tb=5
>>> Algorithm graham burton
>>> Tokenizer osb
>>> PValue bcr
>>> WebStats on
>>> Preference "trainingMode=TEFT" # { TOE | TUM | TEFT |
>>> NOTRAIN }
>>> -> default:teft
>>> Preference "spamAction=tag" # { quarantine | tag | deliver }
>>> ->
>>> default:quarantine
>>> Preference "spamSubject=[SPAM]" # { string } ->
>>> default:[SPAM]
>>> Preference "statisticalSedation=5" # { 0 - 10 } -> default:0
>>> Preference "enableBNR=on" # { on | off } -> default:off
>>> Preference "enableWhitelist=on" # { on | off } -> default:on
>>> Preference "signatureLocation=headers" # { message | headers }
>>> ->
>>> default:message
>>> Preference "tagSpam=off" # { on | off }
>>> Preference "tagNonspam=off" # { on | off }
>>> Preference "showFactors=off" # { on | off } -> default:off
>>> Preference "optIn=off" # { on | off }
>>> Preference "optOut=off" # { on | off }
>>> Preference "whitelistThreshold=10" # { Integer } -> default:10
>>> Preference "makeCorpus=off" # { on | off } -> default:off
>>> Preference "storeFragments=off" # { on | off } ->
>>> default:off
>>> Preference "localStore=" # { on | off } -> default:username
>>> Preference "processorBias=on" # { on | off } -> default:on
>>> Preference "fallbackDomain=off" # { on | off } ->
>>> default:off
>>> Preference "trainPristine=off" # { on | off } -> default:off
>>> Preference "optOutClamAV=off" # { on | off } -> default:off
>>> Preference "ignoreRBLLookups=off" # { on | off } -> default:off
>>> Preference "RBLInoculate=off" # { on | off } -> default:off
>>> Preference "notifications=off" # { on | off } -> default:off
>>> AllowOverride enableBNR
>>> AllowOverride enableWhitelist
>>> AllowOverride fallbackDomain
>>> AllowOverride ignoreGroups
>>> AllowOverride ignoreRBLLookups
>>> AllowOverride localStore
>>> AllowOverride makeCorpus
>>> AllowOverride optIn
>>> AllowOverride optOut
>>> AllowOverride optOutClamAV
>>> AllowOverride processorBias
>>> AllowOverride RBLInoculate
>>> AllowOverride showFactors
>>> AllowOverride signatureLocation
>>> AllowOverride spamAction
>>> AllowOverride spamSubject
>>> AllowOverride statisticalSedation
>>> AllowOverride storeFragments
>>> AllowOverride tagNonspam
>>> AllowOverride tagSpam
>>> AllowOverride trainPristine
>>> AllowOverride trainingMode
>>> AllowOverride whitelistThreshold
>>> AllowOverride dailyQuarantineSummary
>>> AllowOverride notifications
>>> MySQLServer /tmp/mysql.sock
>>> MySQLUser dspam
>>> MySQLPass <redacted>
>>> MySQLDb dspam
>>> MySQLCompress true
>>> MySQLReconnect true
>>> MySQLConnectionCache 10
>>> MySQLVirtualTable dspam_virtual_uids
>>> MySQLVirtualUIDField uid
>>> MySQLVirtualUsernameField username
>>> MySQLUIDInSignature on
>>> HashRecMax 98317
>>> HashAutoExtend on
>>> HashMaxExtents 0
>>> HashExtentSize 49157
>>> HashPctIncrease 10
>>> HashMaxSeek 10
>>> HashConnectionCache 10
>>> Notifications off
>>> PurgeSignatures 14 # Stale signatures
>>> PurgeNeutral 90 # Tokens with neutralish probabilities
>>> PurgeUnused 90 # Unused tokens
>>> PurgeHapaxes 30 # Tokens with less than 5 hits (hapaxes)
>>> PurgeHits1S 15 # Tokens with only 1 spam hit
>>> PurgeHits1I 15 # Tokens with only 1 innocent hit
>>> LocalMX 127.0.0.1
>>> LocalMX 192.168.6.126
>>> SystemLog on
>>> UserLog on
>>> Opt out
>>> ServerMode auto # see postfix.txt
>>> ServerParameters "--deliver=innocent,spam"
>>> ServerDomainSocketPath "/var/spool/postfix/dspam/dspam.sock"
>>> ProcessorURLContext on
>>> ProcessorBias on
>>> StripRcptDomain off
>>> GroupConfig /var/db/dspam/group
>>>
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Dive into the World of Parallel Programming. The Go Parallel
>>> Website,
>>> sponsored by Intel and developed in partnership with Slashdot
>>> Media, is your
>>> hub for all things parallel software development, from weekly
>>> thought
>>> leadership blogs to news, videos, case studies, tutorials and more.
>>> Take a
>>> look and join the conversation now.
>>> http://goparallel.sourceforge.net/
>>> _______________________________________________
>>> Dspam-user mailing list
>>> Dspam-user@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/dspam-user
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming. The Go Parallel
>> Website,
>> sponsored by Intel and developed in partnership with Slashdot Media,
>> is your
>> hub for all things parallel software development, from weekly
>> thought
>> leadership blogs to news, videos, case studies, tutorials and more.
>> Take a
>> look and join the conversation now.
>> http://goparallel.sourceforge.net/
>> _______________________________________________
>> Dspam-user mailing list
>> Dspam-user@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/dspam-user
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media,
> is your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more.
> Take a
> look and join the conversation now.
> http://goparallel.sourceforge.net/
> _______________________________________________
> Dspam-user mailing list
> Dspam-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspam-user
>
> !DSPAM:31,54e0465d101351924013173!
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Dspam-user mailing list
Dspam-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspam-user
