Bug Tracker item #2896879, was opened at 2009-11-12 23:19
Message generated for change (Comment added) made by sbajic
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=2896879&group_id=250683
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: daemon
Group: v3.9.0
Status: Open
Resolution: None
Priority: 9
Private: No
Submitted By: Aria Stewart (aredridel)
Assigned to: Stevan Bajic (sbajic)
Summary: Crash with webui training a fallbackDomain user
Initial Comment:
(gdb) bt
#0 0x000000310987bc70 in strcasecmp () from /lib64/libc.so.6
#1 0x0000000000407458 in ctx_init (ATX=0x7fffbf5e8d40, username=0x0)
at dspam.c:2429
#2 0x000000000040a5bf in process_message (ATX=0x7fffbf5e8d40,
message=0x1005ad0, username=0x0, result_string=0x7fffbf5e8cd8)
at dspam.c:358
#3 0x000000000040c069 in process_users (ATX=0x7fffbf5e8d40, message=0x1001a60)
at dspam.c:1865
#4 0x000000000040cbb8 in main (argc=6, argv=0x7fffbf5e9cd8) at dspam.c:265
this is with a user that's a shared group in the groups file, and is a fallback
user in the preferences.
3.9.0 git
----------------------------------------------------------------------
>Comment By: Stevan Bajic (sbajic)
Date: 2009-11-21 02:43
Message:
> Nov 20 19:56:08 24-75-24-220 dspam[24648]: process_users(): Can
> not fallback to domains for username 'aria.hydra.blumenthals.com'
> without @domain part.
>
Great. Your error report did not let me in peace. I read your report and
read it again and read it again and I just could not find the problem. Then
I started to look deeper into the code of DSPAM and then I saw this
dependency for the @ sign in the user name and then I realized that this is
probably your issue. So changed the code to handle the issue.
> I don't entirely understand what it's saying, but at least it flags it.
> It's implying the username has to be @domain? (Is there any reason this
> has to be the case?)
>
Yes. It is called fallbackDOMAIN. So the domain part is the fall back. If
you have "u...@example.com" and you have "FallbackDomains on" in dspam.conf
then "example.com" is the domain to fall back to. The user name NEEDS a
domain part. Without it DSPAM does not know what domain to take.
It does not matter that you have the domain as shared group. So to sum it
up... assuming the domain you want to fall back to is "blumenthals.com" and
you want to use a shared group for the fallback then:
1) You should have this in your group file:
@blumenthals.com:shared:*...@blumenthals.com
Separating this at the colon means:
1st entry: the GROUP NAME (in your case equal to the fallback domain WITH
the @)
2nd entry: the TYPE OF THE GROUP (in your case a shared group)
3th entry: the MEMBERS OF THE GROUP (in your case everyone having
@blumenthals.com in his/her username)
2) Ensuring that "blumenthals.com" is a fallback domain you need to enable
"FallbackDomains on" in your dspam.conf and you need to have a user with
the name "@blumenthals.com" (the domain WITH a @ in front) in the
preference table with the preference "fallbackDomain=on"
3) Each user you want to participate in this fallback domain scenario
needs to have @lumenthals.com in his DSPAM user name else the the whole
fallback domain stuff IS NOT going to work.
My patch from yesterday is handling all of this and should the user name
not have a @domain part then now DSPAM is not honoring the "fallbackDomain"
preference nor is it honoring the "FallbackDomains on" setting from
dspam.conf. Basically it just ignores them and continues operating as if
those two settings would have never been turned on (since DSPAM can't fall
back to the domain part of the user name because it can't determine the
domain the user uses).
Under a proper configuration you could go on and have u...@blumenthals.com
and process any mail for that user with DSPAM without having opened an
account for that user in DSPAM. And this is still working. Even if you
would run DSPAM in OptIn mode and not have OptIn active for
u...@blumenthals.com, it would work. Regardless the fact that you never
opted in the u...@blumenthals.com account. That would normally not be
possible. Without fallbackDomains that user would not be processed. But now
with my patch from yesterday things change. If you have made an error with
your configuration then DSPAM will continue as usual and try to use the
full user name as if there have not been any FallbackDomains settings in
dspam.conf or any failbackDomain settings in the preference extension.
And to continue to explain my example from above: If you would have made
an error in the configuration then DSPAM will now log that issue. And if
you would have turned OptIn on (so that every user would need to opt into
DSPAM processing) and for what ever reason you would try to process a user
where DSPAM can not extract the domain part from the user name then DSPAM
will continue without and then see that the user in question is not opted
into DSPAM filtering and then it would not process that message for that
user. With fallback domains turned on it would first try to get the domain
name and look if the domain name is used as a fallback domain and if that
would be the case then process the mail for the user not under his name but
under the domain name he has. And that would then work around the issue of
that the user is not opted into DSPAM filtering (since the domain is opted
into DSPAM filtering but again... the user does not need to be if fallback
domains is active).
Can I now close this bug here?
--
Kind Regards from Switzerland,
Stevan Bajić
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-21 01:59
Message:
It also /does/ retrain even so, now, so I could theoretically throw this
into production. What's it not doing that it should, if it is in fact
retraining the message?
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-21 01:58
Message:
Ah, excellent. I do!
Nov 20 19:56:08 24-75-24-220 dspam[24648]: process_users(): Can not
fallback to domains for username 'aria.hydra.blumenthals.com' without
@domain part.
I don't entirely understand what it's saying, but at least it flags it.
It's implying the username has to be @domain? (Is there any reason this has
to be the case?)
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-21 01:48
Message:
This here is not going to work with fallback domains:
--user aria.hydra.blumenthals.com
I submitted yesterday a bunch of patches dealing with that in the proper
way since I suspected something to happen in your setup and I just wanted
to handle that and print out a message informing the user that he/she is
doing wrong. Here is the submitted code:
http://dspam.git.sourceforge.net/git/gitweb.cgi?p=dspam/dspam;a=commit;h=5183294f11e985b242d3817ae47eef9f1d8e9d2c
Do you get another message if you compile DSPAM again from GIT and re-run
the test?
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-21 01:39
Message:
That's running on the commandline, but the same command that dspam.cgi
executes with system()
> How do you run the CGI's in the WebUI (SuEXEC? Other way?)
Directly as CGI. apache is in dspam.conf's Trust list.
- With what user?
domain.com -- the domain user. It's got an entry for @domain.com with the
same uid in dspam_virtual_uids. If I don't, the web UI won't show the
history for the user.
- With what user/group does the WebUI run?
apache
- Can you post the output of "ls -lah" against the dspam binary or post
the output of: ls -lah $(whereis dspam|awk '{print $2}')
-rwx--s--- 1 dspam mail 274K Nov 19 00:06 /usr/bin/dspam
dspam: /usr/bin/dspam /etc/dspam /usr/share/dspam
/usr/share/man/man1/dspam.1.gz
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-21 01:25
Message:
The errors you posted are when executing the commands from the command line
or when executing from within the WebUI?
Can you answer the following questions:
- How do you run the CGI's in the WebUI (SuEXEC? Other way?)
- With what user?
- With what user/group does the WebUI run?
- Can you post the output of "ls -lah" against the dspam binary or post
the output of: ls -lah $(whereis dspam|awk '{print $2}')
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-21 01:15
Message:
Pardon. I mean that that fixes the asking the wrong question problem, but
leaves the greater problem -- not being able t train from the webui.
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-21 01:13
Message:
I had that in my dspam.conf, same issue.
I've tried with pgsql (now that the compilation bug is fixed, I got that
working) -- same issue. Looks like it's not in any of the db-specific
code:
dspam --source=error --class=innocent
--signature=3,4b072ebc91645760821076 --user aria.hydra.blumenthals.com
gives me
Nov 20 19:08:05 24-75-24-220 dspam[9489]: Unable to determine the
destination user
Nov 20 19:08:05 24-75-24-220 dspam[9489]: Context initialization failed
Nov 20 19:08:05 24-75-24-220 dspam[9489]: process_message returned error
-2. dropping message.
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-20 21:28
Message:
Please add in your dspam.conf this here:
MySQLUIDInSignature on
Restart the daemon and then try again.
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-20 16:53
Message:
Things are added to the database with uid = 4, signature =
4b06b9f741961536915273, for example, but if I strace dspam when retraining
with --signature=4,4b06b9f741961536915273, I see that dspam makes this
query:
write(3, "b\0\0\0\3SELECT data,length FROM dspam_signature_data WHERE
uid=4 AND signature=\"4,4b06b9f741961536915273\"", 102) = 102
Any idea why it'd ask the wrong question?
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-17 19:52
Message:
[r...@spamfilter ~]# dspam --version
DSPAM Anti-Spam Suite 3.9.0-BETA4 (agent/library)
Copyright (c) 2002-2009 DSPAM Project
http://dspam.sourceforge.net.
DSPAM may be copied only under the terms of the GNU General Public
License,
a copy of which can be found with the DSPAM distribution kit.
Configuration parameters: '--host=x86_64-redhat-linux-gnu'
'--build=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux'
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
'--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64'
'--libexecdir=/usr/libexec' '--localstatedir=/var'
'--sharedstatedir=/usr/com' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--sysconfdir=/etc/dspam'
'--with-dspam-home=/var/spool/dspam' '--with-logdir=/var/log/dspam'
'--enable-shared' '--enable-static' '--enable-daemon'
'--enable-external-lookup' '--enable-clamav' '--enable-debug'
'--enable-syslog' '--enable-long-usernames' '--enable-domain-scale'
'--enable-virtual-users'
'--with-storage-driver=hash_drv,mysql_drv,pgsql_drv'
'--with-mysql-includes=/usr/include/mysql'
'--with-mysql-libraries=/usr/lib64/mysql' 'CFLAGS=-O2 -g -m64
-mtune=generic' 'CXXFLAGS=-O2 -g -m64 -mtune=generic' 'FFLAGS=-O2 -g -m64
-mtune=generic' 'build_alias=x86_64-redhat-linux-gnu'
'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux'
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-17 19:51
Message:
dspam.conf:
Home /var/spool/dspam
StorageDriver /usr/lib64/dspam/libmysql_drv.so
DeliveryHost 127.0.0.1
DeliveryPort 10025
DeliveryIdent localhost
DeliveryProto SMTP
FallbackDomains on
OnFail error
Trust root
Trust dspam
Trust apache
Trust mail
Trust mailnull
Trust smmsp
Trust daemon
Debug *
DebugOpt process spam fp classify
ClassAliasSpam spam
TrainingMode teft
TestConditionalTraining on
Feature whitelist
Algorithm graham burton
Tokenizer chain
PValue bcr
WebStats on
ImprobabilityDrive on
Preference "spamAction=quarantine"
Preference "signatureLocation=message"
Preference "showFactors=on"
Preference "fallbackDomain=on"
AllowOverride trainingMode
AllowOverride spamAction spamSubject
AllowOverride statisticalSedation
AllowOverride enableBNR
AllowOverride enableWhitelist
AllowOverride signatureLocation
AllowOverride showFactors
AllowOverride optIn optOut
AllowOverride whitelistThreshold
AllowOverride dailyQuarantineSummary
AllowOverride fallbackDomain
MySQLServer /var/lib/mysql/mysql.sock
MySQLUser dspam
MySQLPass redacted
MySQLDb dspam
MySQLReconnect true
MySQLConnectionCache 10
PgSQLServer 127.0.0.1
PgSQLPort 5432
PgSQLUser dspam
PgSQLDb dspam
PgSQLConnectionCache 3
PgSQLUIDInSignature on
HashRecMax 98317
HashAutoExtend on
HashMaxExtents 0
HashExtentSize 49157
HashPctIncrease 10
HashMaxSeek 10
HashConnectionCache 10
Notifications on
PurgeSignatures 14
PurgeNeutral 90
PurgeUnused 90
PurgeHapaxes 30
PurgeHits1S 15
PurgeHits1I 15
LocalMX 127.0.0.1
SystemLog on
UserLog on
Opt out
ParseToHeaders on
ChangeModeOnParse on
ChangeUserOnParse full
ClamAVPort 3310
ClamAVHost 127.0.0.1
ClamAVResponse reject
ServerDomainSocketPath /var/run/dspam/dspam.sock
ServerMode auto
ServerQueueSize 32
ServerMode auto
ServerPass.Relay1 redacted
ServerParameters "--deliver=innocent -d %u"
ServerIdent "spamfilter.blumenthals.com"
ClientHost /var/run/dspam/dspam.sock
ClientIdent redac...@relay1
ProcessorURLContext on
ProcessorBias on
StripRcptDomain off
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-17 19:33
Message:
Well, it logs and stops the segfault with that patch. Now to determine why
it's null ...
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-17 19:18
Message:
Looking at it now. Still happens with current git, but I'm trying other
usernames in the group to see if that's the problem.
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-15 23:41
Message:
Hallo Aredridel,
do you have any update on this case?
--
Kind Regards from Switzerland,
Stevan Bajić
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-13 12:48
Message:
Hallo Aredridel,
when looking at your stack trace I see that ctx_init() get's called with
an NULL username. This should NOT happen. I could trap that case in
ctx_init(). Better: I must trap that case. Can you apply this patch to the
latest GIT source and try if the crash still presists:
------------------------
diff --git a/src/dspam.c b/src/dspam.c
index 7bfe301..b97f4a5 100644
--- a/src/dspam.c
+++ b/src/dspam.c
@@ -1,4 +1,4 @@
-/* $Id: dspam.c,v 1.378 2009/11/13 02:26:19 sbajic Exp $ */
+/* $Id: dspam.c,v 1.379 2009/11/13 12:44:59 sbajic Exp $ */
/*
DSPAM
@@ -2367,6 +2367,12 @@ NEXT:
*/
DSPAM_CTX *ctx_init(AGENT_CTX *ATX, const char *username) {
+
+ if (username == NULL) {
+ LOG (LOG_CRIT, ERR_AGENT_USER_UNDEFINED);
+ return NULL;
+ }
+
DSPAM_CTX *CTX;
char filename[MAX_FILENAME_LENGTH];
char ctx_group[128] = { 0 };
------------------------
I would say that the crash is gone but you will have a run time error
because DSPAM can not work on a undefined user name.
Kind Regards from Switzerland,
Stevan Bajić
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-13 12:34
Message:
Hallo Aredridel,
I can not reproduce this here. What I have done so far:
dspam.conf:
FallbackDomains on
MySQL:
insert into dspam_virtual_uids (uid,username) values
(null,'@domain.com');
Shell (I run a DSPAM in OptIn mode):
dspam_admin change preference @domain.com fallbackDomain on
dspam_admin change preference @domain.com optOut off
dspam_admin change preference @domain.com optIn on
After that my data in MySQL looks like this:
dspam_virtual_uids:
+------+-------------+
| uid | username |
+------+-------------+
| 2087 | @domain.com |
+------+-------------+
dspam_preferences:
+------+----------------+-------+
| uid | preference | value |
+------+----------------+-------+
| 2087 | fallbackDomain | on |
| 2087 | optIn | on |
| 2087 | optOut | off |
+------+----------------+-------+
group file:
@domain.com:shared:*...@domain.com
Forced a learning for a bunch of messages. All of them resulted in the
same output:
nyx ~ # dspam --user @domain.com --deliver=summary --stdout
--source=corpus --class=spam < path/to/message1.txt
X-DSPAM-Result: @domain.com; result="Spam"; class="Spam";
probability=1.0000; confidence=1.00; signature=N/A
nyx ~ #
Tokens where added to the MySQL table:
> select count(*) from dspam_token_data where uid=2087;
+----------+
| count(*) |
+----------+
| 36377 |
+----------+
Now did a test with a user that does NOT exist in dspam_virtual_uids:
nyx ~ # dspam --user t...@domain.com --deliver=summary --stdout --process
< path/to/message1000.txt | grep "^X\-DSPAM\-"
X-DSPAM-Result: @domain.com; result="Spam"; class="Spam";
probability=1.0000; confidence=0.99; signature=2087,4afd3dd512221635636504
X-DSPAM-Result: Spam
X-DSPAM-Processed: Fri Nov 13 12:07:01 2009
X-DSPAM-Confidence: 0.9899
X-DSPAM-Improbability: 1 in 9809 chance of being ham
X-DSPAM-Probability: 1.0000
X-DSPAM-Signature: 2087,4afd3dd512221635636504
nyx ~ #
Looks okay to me. The uid used for that user is the one from the user
"@domain.com". Okay. Now let's try to retrain that message:
nyx ~ # dspam --user t...@domain.com --stdout --source=error
--class=innocent --signature=2087,4afd3dd512221635636504
nyx ~ #
Works. I see in the Web-UI that the message got changed. Now retrain it
back (undo) to spam:
nyx ~ # dspam --user t...@domain.com --stdout --source=error --class=spam
--signature=2087,4afd3dd512221635636504
nyx ~ #
Works as well.
The only difference I see in your and my setup is that I used "@domain" as
the group name for the shared group. You used "domain.com" but "domain.com"
does not exist as user in the dspam_virtual_uids table so I used
"@domain.com". Anyway... going back to just use "domain.com" for the group
(@domain.com:shared:*...@domain.com) lead in the same result. Except that this
time the result from DSPAM was innocent instead of spam but that is
probably because the used user "domain.com" for the shared group does not
exist (only @domain.com exists as an user):
nyx ~ # dspam --user t...@domain.com --deliver=summary --stdout --process
< path/to/message1000.txt | grep "^X\-DSPAM\-"
X-DSPAM-Result: @domain.com; result="Innocent"; class="Innocent";
probability=0.0000; confidence=0.70; signature=2087,4afd3f6b15271305324895
nyx ~ #
==> The strange thing here is that I get a summary result but "--stdout"
should FORCE the whole message to be displayed to the console (incl. a
summary). Something is strange here. I can however force deliver to show up
on screen by instructing DSPAM to deliver innocent or spam mails (which
should have been selected by the "--stdout" switch already):
nyx ~ # dspam --user t...@domain.com --deliver=innocent,spam --stdout
--process < path/to/message1000.txt | grep "^X\-DSPAM\-"
X-DSPAM-Result: Innocent
X-DSPAM-Processed: Fri Nov 13 12:22:48 2009
X-DSPAM-Confidence: 0.6701
X-DSPAM-Improbability: 1 in 204 chance of being spam
X-DSPAM-Probability: 0.0000
X-DSPAM-Signature: 2087,4afd418820052092820584
nyx ~ #
--> BINGO! That is the right output. It should have been that output from
the beginning. Anyway... this is another issue I need to look after later.
I went ahead and retrained that message to be spam:
nyx ~ # dspam --user t...@domain.com --stdout --source=error --class=spam
--signature=2087,4afd3f6b15271305324895
nyx ~ #
The Web-UI changed and the message was switched to spam.
What have I done on my setup that you have not? Can you attach your
dspam.conf and post the output of "dspam --version"?
Kind Regards from Switzerland,
Stevan Bajić
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-13 07:29
Message:
I do have the rest as you suggested.
----------------------------------------------------------------------
Comment By: Aria Stewart (aredridel)
Date: 2009-11-13 00:40
Message:
I have:
domain.com:shared:*...@domain.com
The docs were a little unclear how it all fits together, the username
munging, etc.
----------------------------------------------------------------------
Comment By: Stevan Bajic (sbajic)
Date: 2009-11-12 23:38
Message:
Hallo Aredridel,
how can you mirror that setup on my end? What do I need to have in the
group file and what in the preference extension?
If I have understood your setup then it's +/- like this:
group file:
@domain.com:shared,managed:*...@another.com
preferences for domain.com:
fallbackDomain=on
dspam.conf:
FallbackDomains on
Right?
Kind Regards from Switzerland,
Stevan Bajic
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=1126467&aid=2896879&group_id=250683
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Dspam-devel mailing list
Dspam-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspam-devel
